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FOREWORD 


One  of  the  research  goals  of  the  Decision  Sciences  Laboratory 
Is  the  development  of  design  principles  for  automated  training  sub¬ 
systems  which  could  be  built  into  future  Information  Systems.  Such 
subsystems  would  provide  Information  Systems  with  the  capability  of 
automatically  training  their  own  operators.  The  need  for  such  on- 
the-job  training  capability  has  already  become  apparent.  To  be  able 
to  design  such  a  capability  requires  first  the  solution  of  many  con¬ 
ceptual  and  experimental  problems. 

Task  768204,  Automated  Training  for  Information  Systems,  under 
Project  7682,  Man-Computer  Information  Processing,  was  established  to 
formulate  and  answer  some  of  these  questions. 

This  report  is  one  in  a  planned  series  supporting  Task  768204. 

The  project  was  undertaken  by  Decision  Sciences  Laboratory  in  support 
of  the  473L  Systems  Program  Office.  Dr.  Sylvia  R.  Mayer  of  Decision 
Sciences  Laboratory  served  as  Air  Force  Task  Scientist  and  contract 
monitor. 

Contract  support  was  provided  by  the  American  Institutes  for 
Research,  Pittsburgh,  Pennsylvania,  under  Contract  No.  AF  19(628 ) -2535 > 
with  Mr.  Harris  H.  Shettel,  principal  investigator;  Mrs.  Doris  Clapp 
Slough,  project  director;  Mr.  David  P.  Yens;  and  Miss  Judi  L.  Northrup . 

The  technical  guidance  ana  support  provided  by  Lt .  Colonel  Wood 
Ellis  of  the  USAF  Command  Post  are  gratefully  acknowledgec. .  Thanks 
are  also  extended  to  the  personnel  of  the  USAF  Command  Post  who  served 
in  the  field  tests  of  this  course. 

In  ensuring  the  development  of  the  operational  specification  for 
computer  directed  training  in  accordance  with  approved  473L  standaros, 
the  A.I.R.  project  staff  had  the  extremely  valuable  guidance  of 
Dr.  William  F.  Bennett,  Mr.  Marvin  L.  Chenevert,  and  Mr.  Jack  D. 

Schiff  of  the  Federal  Systems  Division  of  International  Business 
Machines,  Inc.  This  is  gratefully  acknowledged.  In  addition,  helpful 
contributions  to  the  training  design  for  the  computer  directed  training 
package  were  made  by  Lt .  Colonel  Wood  Ellis,  Dr.  Sylvia  R.  Mayer, 

Mr.  Jack  D.  Schiff,  and  Mr.  Marvin  L.  Chenevert. 

This  technical  report  has  been  reviewed  and  is  approved. 


ROY  MORGAN 
Colonel,  USAF 
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ABSTRACT 


This  report  summarizes  the  development  and  evaluation  of  a 
programed,  self -instructional  course  for  on-the-job  training  of 
Air  Staff  personnel  in  the  use  of  Intermediate  Query  Language, 

Model  II.  This  is  an  information  retrieval  language  used  with 
the  computer  based,  Air  Force  command  and  control  system.  System 
473L.  In  addition.  It  describes  a  computer  directed  training 
capability  that  was  designed  specifically  to  use  System  473L  itself 
to  effectively  and  efficiently  provide  training  in  Query  Language. 
The  report  describes  the  need  for  on-the-job  training  and  the 
rationale  for  a  computer  directed  training  capability  to  provide 
this  training.  It  describes  the  development  of  the  programed  text, 
^he  text  itself,  and  the  effectiveness  of  the  text  materials  based 
on  tryout  data.  Finally,  a  description  of  the  proposed  computer 
directed  training  course  is  given,  with  emphasis  on  the  training 
design.  The  473L  System  configuration  using  the  AN/FYQ-11  computer, 
towards  which  this  study  was  oriented,  will  not  be  implemented  for 
the  Headquarters  U.  S.  Air  Force  Command  and  Control  System.  How¬ 
ever,  this  design  study  for  the  training  subsystem  may  be  of  Inter¬ 
est  to  researchers  on  computer-directed  Instructional  systems. 


iii 


TAFLE  OF  CONTENTS 


Page 

FOREWORD .  ii 

ABSTRACT . ill 

SECTION  I .  INTRODUCTION 

1.1  Background .  1 

1.2  The  Air  Force  Command  and  Control  System  (473L)  .  1 

1.3  The  Training  Problem  .  2 

1.4  The  Overall  Training  Strategy  .  2 

1.4.1  The  Need  for  a  Programed,  Self -Instructional 

Text .  3 

1.4.2  The  Need  for  an  Operational  Specification  for 

Computer  Directed  Training  .  3 

SECTION  II.  THE  DEVELOPMENT  OF  THF  PROGRAMED  TEXT 

2.1  The  Selection  of  Course  Objectives  and  Content  .  7 

2.2  The  Selection  of  Specific  Training  Strategies  .  .  7 

2.2.1  Trainee  Characteristics  .  7 

2.2.2  The  Course  Objectives  .  8 

2.2.3  Characteristics  of  the  Course  Content  .  8 

2.3  Description  of  the  Final  Program  .  9 

2.3.1  The  Preprogram .  9 

2.3.2  The  Textual  Program  on  Query  Language  .  9 

2. 3. 2.1  Number  of  Frames  .  9 

2. 3. 2. 2  Content  of  Frames  .  9 

2. 3. 2. 3  Reference  Materials  .  12 

2. 3. 2. 4  Program  Format  .  22 

2.3.3  Computer  Exercises  .  22 

2.3.4  Test . 22 

2.4  The  Drafting  of  Frames,  Initial  Program  Tryouts, 

Technical  Review,  and  Revisions  .  24 

2.4.1  Subjects . 24 

2.4.2  Results . 24 

2.5  Field  Tryout  and  Revision . 26 

2.5.1  Design . 26 

2.5.2  Subjects . 26 

2.5.3  Administration  of  the  Field  Tryout . 27 

2.5.4  Results  of  the  Field  Tryout . 27 

2. 5. 4.1  Completion  Time3  for  the  Programed  Materials  27 

2. 5. 4. 2  Scores  on  the  Final  Posttest  .  28 

2.5.5  Revisions  After  the  Field  Tryout  .  28 

2.6  Discussion  and  Recommendations  .  29 

SECTION  III.  DEVELOPMENT  OF  THE  OPERATIONAL  SPECIFICA¬ 
TION  FOR  COMPUTER  DIRECTED  TRAINING 

3.1  Introduction .  31 

3.2  Development  of  the  Operational  Specification  .  .  31 

3.3  Description  of  the  Operational  Specification  .  .  32 

3.3.1  The  Training  Sequence  Logic  .  32 

3. 3. 1.1  Complexity  of  the  Training  Design .  32 


iv 


Table  of  Contents  (cont.) 


Page 


3. 3. 1.2  Provision  for  Adaptation  of  the  Capability  .  33 


3. 3. 1*3  The  Training  Sequence  Logic . 33 

3.3.2  Operating  Procedures  and  the  Procedural  Flow 

Diagram . 45 

3.4  Discussion  and  Recommendations  .  49 

REFERENCES . 51 


APPENDICES 

A  -  Number  of  Frames  in  Each  Volume  of  the  Program 
B  -  Final  Test 

C  -  Final  Test  Scoring  and  Answer  Key 

D  -  Sequence  and  Contents  of  the  Computer  Directed  Training  Sets 


TAELES 

1  -  Average  Error  Rates  on  Draft  Volumes  Pertaining  to 

QL  that  Were  Used  in  the  Preliminary  Tryouts  ....  25 

2  -  Percent  Error  for  Each  Trainee  on  the  Program  and 

Percent  Correct  on  the  Test  for  Each  of  the 
Preliminary  Tryouts  .  25 

3  -  Background  Data  for  Trainees  Participating  in  the 

Field  Tryout . 26 

4  -  Program  Completion  Time  for  Each  Trainee . 27 

5  -  Percent  Correct  Scores  on  Final  Test  for  Individual 

Trainees  in  the  Field  Tryout . 28 

6  -  Design  for  Computer-Directed  Training  Overlay  Process 

Step  Key  Functions . 46 


FIGURES 

1  -  Schematic  Diagram  of  Planned  Information  Flow  in  the 

Design  for  a  Second -Generation  473L  System  .  14 

2  -  A  description  of  the  Proposed  Procedural  Steps  Required 

to  Retrieve  Data  for  a  Specified  Problem  Using  Query 
Language . 15 

3  -  A  Partial  Listing  and  Description  of  the  Data  Files  .  17 

4  -  An  Incomplete  Table  with  Typical  Values  .  18 


v 


Table  of  Contents  (cont.) 

Page 

FIGURES  (cont.) 

5  -  An  Example  of  How  Information  Is  to  be  Retrieved  from  the 

Data  Files  Using  Query  Language  .  19 

6  -  Formats  and  Functions  of  SUM  in  the  Output  Selector  .  .  20 

7  -  Examples  of  Input  and  Output  Formats  for  SUM . 21 

8  -  The  Training  Sequence  Logic  Flow  Diagram  for  Computer 

Directed  Training  .  40 


vi 


Editor's  Note 


The  reader  should  note  that  the  473L  System  configuration 
using  the  AN/FYQ-11  computer,  towards  which  this  study  was  ori¬ 
ented,  will  not  be  Implemented  for  the  Headquarters  U.  S.  Air 
Force  Command  and  Control  System. 

This  report  Is  presented  since  the  general  design  of  the 
training  subsystem  could  serve  as  a  model  for  other  computer- 
directed  training  courses  In  other  computer-based  military  Infor¬ 
mation  systems. 


v  1 1 


Section  I 


INTRODUCTION 


1.1  BACKGROUND 

Query  Language  is  a  constrained  version  of  the  English 
language  which  has  been  developed  as  a  mode  of  man-machine 
communication  in  the  Air  Force  Command  and  Control  System, 

System  473L  (this  system  is  described  in  section  1.2).  Query 
Language  has  evolved  over  a  period  of  time  to  accommodate 
projected  changes  in  the  system  hardware  (viz.,  from  the  IBM 
1410  to  the  Librascope  3055 )>  and  to  permit  problem  solutions 
for  additional  areas  of  resource  management.  This  evolution 
involved  three  successive  versions  of  the  language,  in  the  fol¬ 
lowing  order:  OTC  Query  Language;  Query  Language  Model  I; 
and  Query  Language  Model  II.  The  modifications  in  the  language 
were  substantial  enough  so  that  personnel  trained  in  the  use 
of  the  Query  Language  designed  for  the  Operational  Training 
Capability  (OTC)  phase  of  System  473L  could  not  easily  transi¬ 
tion  to  the  use  of  the  Query  Language  designed  for  the  Model  II 
phase  of  System  473L.  In  addition,  appropriate  training  was 
needed  for  new  personnel.  Therefore,  a  new  training  package 
was  requested  to  teach  the  version  of  Query  Language  appropriate 
to  the  Model  II  phase  of  System  473L.  It  was  further  requested 
that  a  detailed  description  be  developed  for  a  computer  directed 
training  capability,  in  accord  with  473L  standards,  so  that  the 
Air  Force  could  evaluate  the  feasibility  of  implementing  such  a 
capability . 

This  report  describes  the  development  of  a  programed,  self- 
instruct lonal  course  designed  for  on-the-job  training  of  System 
473L  users  In  Intermediate  Query  Language,  Model  II.1  In  addi¬ 
tion,  It  describes  the  development  of  an  operational  specifica¬ 
tion  for  a  computer  directed  training  (CDT)  capability  designed 
specifically  to  use  the  473L  System  itself  to  effectively  and 
efficiently  train  Air  Staff  personnel  in  the  use  of  Intermediate 
Query  Language,  Model  II.  The  proposed  computer  directed 
training  capability  has  further  significance  in  its  potential, 
with  appropriate  adaptations,  for  providing  training  in  other 
uses  of  System  473L.  In  addition,  the  general  design  of  the 
training  course  set  forth  in  the  operational  specification, 
i.e.,  the  training  sequence  logic,  could  serve  as  a  model  for 
similar  computer  directed  training  courses  in  other  computer 
based  military  systems. 

1.2  THE  AIR  FORCE  COMMAND  AND  CONTROL  SYSTEM  (473L) 

System  473L  is  an  information  processing  and  retrieval 
system  located  in  the  Air  Force  Command  Post  at  the  Pentagon. 

An  integral  part  of  the  system  is  a  large  capacity  computer. 

^The  scope  of  Intermediate  Query  Language,  Model  II,  is  defined 
in  Chapter  3  of  473L-0S-40:  Operational  Specification  for 
Que_ry;  Language,  Model  II,  dated  13  April  1965,  Unclassified. 


The  system  is  designed  for  use  by  Air  Staff  personnel  in 
solving  USAF  resource  management  problems.  There  are  two 
basic  methods  of  communication  between  the  System  473L 
operator  and  the  data  processing  subsystem:  l)  operational  capability 
overlays  and  2)  Query  Language.  With  the  overlay  capabilities 
the  operator  is  guided  in  selecting  and  making  his  input  but 
his  retrieval  of  information  is  restricted  to  a  set  of  pre¬ 
viously  specified  outputs.  With  Query  Language,  the  operator's 
input  is  not  guided,  but  his  retrieval  of  information  is  not  as 
restricted.  Thus,  Query  Language  is  a  vital  complement  to  the 
overlay  method  of  communication  with  System  473L. 

1.3  THE  TRAINING  PROBLEM 

In  developing  an  overall  training  strategy,  consideration 
was  given  to  two  interacting  aspects  of  the  training  problem: 
the  training  task  itself  and  the  trainees. 

The  use  of  System  473L  is  not  restricted  to  a  small  number 
of  system  operators;  it  is  intended  that  this  system,  including 
Query  Language,  be  used  by  a  large  number  of  Air  Staff  personnel. 

These  constituted  the  potential  trainees  for  this  course. 

It  was  specified  that  the  training  course  should  teach 
only  Intermediate  Query  Language,  since  it  includes  those 
aspects  of  the  language  that  are  most  commonly  used  by  Air 
Staff  personnel.  The  more  advanced  uses  of  Query  Language  are 
restricted  primarily  to  programmer  use  (e.g.,  for  maintaining 
and  updating  the  data  base),  and  thus  these  uses  were  not  con¬ 
sidered  appropriate  for  the  proposed  course. 

Proficient  use  of  Intermediate  Query  Language  requires  a 
knowledge  of  both  the  structure  and  contents  of  the  system 
data  base  and  the  rules  regulating  the  use  of  vocabulary, 
syntax,  grammar,  and  punctuation  of  Query  Language.  In  addi¬ 
tion,  since  Query  Language  must  be  entered  on  the  System  473L 
integrated  console,  the  user  must  ultimately  become  proficient 
in  the  operation  of  this  console.  Thus,  while  training  on  tne 
basic  use  of  the  console  is  not  an  integral  part  of  this  train¬ 
ing  course,  it  was  considered  desirable  to  incorporate  practice 
on  the  operations  that  are  most  frequently  used  to  input  Query 
Language  . 

1.4  THE  TRAINING  STRATEGY 

Due  to  the  dynamic  structure  and  large  size  of  the  data 
base  for  System  4/3L,  the  dynamic  nature  of  Query  Language 
itself,  the  frequent  turnover  of  Air  Staff  personnel,  and  the 
vagaries  of  their  normal  and  emergency  duty  schedules,  effec¬ 
tive  and  efficient  training  of  Air  Staff  personnel  as  profi¬ 
cient  users  of  Query  Language  requires  an  on-the-job  training 
capability . 
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1.4.1  The  Need  for  a  Programed,  Self -Instructional  Text.  The 
need  for  maximum  flexibility  in  the  required  training  schedule 
to  adapt  to  the  needs  of  individual  trainees,  in  terms  of  both 
individual  learning  rates  and  individual  schedules  of  Air  Force 
duties,  suggested  the  desirability  of  developing  programed,  self- 
instructional  materials  that  could  be  used  for  on-the-job  training. 

1.4.2  The  Need  for  an  Operational  Specification  for  Computer 
Directed  Training.  Since  System  473L  is  an  information  processing 
system,  it  is  capable  of  implementing  both  training  and  evaluation 
functions.  The  desirability  of  using  the  system  itself  as  an 
adaptive  training  device,  in  conjunction  with  programed  training 
materials,  was  suggested  by  several  considerations: 

1)  The  design  for  the  second -generation  System  473L  has  certain 
physical  characteristics  that  make  it  especially  amenable  to  perform¬ 
ing  training  and  evaluation  functions.  The  computer's  integrated 
console  contains  a  keyboard  to  permit  operator  inputs  and  an  overlay 
board  to  permit  access  to  the  internal  program.  The  overlay  i3 
especially  useful,  since  a  special  panel  can  be  cesigned  to  provide 
the  trainee  with  direct  access  to  and  control  over  the  computer 
directed  training  functions.  The  computer  can  be  programmed  to 
evaluate  trainee  inputs.  In  addition,  the  integrated  console  has  a 
cathode  ray  tube  output  that  permits  alphanumeric  displays.  Such 
displays  can  be  used  to  present  both  instructional  (cue)  and  evalua¬ 
tive  (test)  materials.  Further,  the  system  permits  time -sharing , 

so  that  more  than  one  training  station  could  be  made  available  at 
any  one  time; 

2)  Letting  the  trainee  use  tne  system  itself  for  training 
would,  it  is  believed,  make  the  training  materials  inherently  more 
interesting  and,  as  a  result,  increase  tne  trainee's  motivation  to 
complete  the  course.  It  should  also  enhance  the  trainee's  profi¬ 
ciency  in  performing  specific  system  operations.  Enhanceo  facility 
in  operating  the  system  would  probably  increase  the  frequency  with 
which  trainees  would  use  the  system  in  the  future; 

3)  The  design  for  the  System  473L  computer  permits  more 
flexibility  than  conventional  training  materials  in  adapting  the 
content  and  sequence  of  instruction  to  fit  individual  needs. 

Remedial  sequences  and  content  can  be  contingent  on  an  evaluation 
of  a  series  of  criterion  responses; 

4)  In  addition,  a  computer  can  be  more  flexibly  responsive 
than  a  text  in  complying  with  trainee  requests  for  particular 
training  materials.  That  is,  a  computer  is  more  adaptable  to 
learner-controlled  instruction.  This  is  a  significant  advantage 
not  only  for  the  training  phase  of  a  course,  but  more  especially 
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for  proficiency  maintenance  training,  when  the  trainee  is  more 
capable  of  self-evaluation  and  determination  of  necessary  remedial 
materials ; 

5)  Not  only  can  knowledge  of  results  be  immediate,  but 
analysis-feedback  can  be  provided  to  the  trainee  based  on  his 
specific  past  performance; 

6)  Due  to  the  computer's  planned  capability  for  rapid  data- 
analysis  and  its  large  storage  capacity,  detailed  recording  and 
analysis  of  trainee  responses  and,  therefore,  increased  accuracy 

in  assessing  trainee  proficiency  and  determining  needs  for  specific 
revisions  in  the  training  materials,  is  more  feasible  than  with 
conventional  materials; 

7)  The  potential  high-speed  printout  capability  provides  auto¬ 
matic  records  of  student  progress; 

8)  A  further,  potentially  significant  advantage  of  a  computer 
directed  training  capability  is  that,  with  some  adaptations,  it 
could  also  be  used  to  provide  training  in  other  uses  of  System 
473L. 


Pretest  data  from  the  tryouts  conducted  with  the  programed 
text  for  OTC  Query  Language2  indicated  that  the  potential  trainees 
are  relatively  homogeneous  in  terms  of  their  prior  knowledge  of 
Query  Language,  e.g.,  on  the  pretest  used  for  the  initial  tryout, 
no  trainee  scored  higher  than  three  per  cent.  Therefore,  com¬ 
puter  directed  training  was  considered  most  desirable  for  the 
final  phases  of  training,  when  there  would  be  a  greater  need 
for  instructional  flexibility  due  to  increased  variability  in 
trainee  proficiency.  However,  it  was  felt  that  the  increased 
motivation  for  training  and  the  projected  increase  in  future 
use  of  the  system  due  to  increased  operator  proficiency  would 
justify  the  additional  expenditure  necessary  to  provide  computer 
direction  for  all  phases  of  training,  provided  that  this  training 
capability  would  not  impeoe  the  operational  capabilities  of 
System  473L. 

In  view  of  the  above  advantages  of  a  computer  directed 
training  capability,  the  Air  Force  cecided  that  a  detailed 
operational  specification  should  be  developed  to  describe  such 
a  capability  and  this  operational  specification  should  then  be 
evaluated  to  assess  the  feasibility  of  implementation  in  terms 
of  the  impact  on  System  4f3L  operational  capabilities  anc  the 
scope  of  programming  required  for  this  capability.  Since  the 
need  for  the  programed,  self -instructional  text  was  immediate 
and  the  feasibility  of  computer  directed  training  was  uncertain. 


^A  description  of  the  development  and  evaluation  of  the  programed 
text  for  OTC  Query  Language  is  presented  in  reference  1.  The 
text  itself  is  presented  in  reference  2. 
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the  Air  Force  indicated  that  the  programed  instructional  text 
materials  should  be  developed  first  and  that  they  should  consti¬ 
tute  a  self-sufficient  package,  not  dependent  on  any  aspect  of 
the  computer  directed  training  package. 
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Section  II 


THE  DEVELOPMENT  OF  THE  PROGRAMED  TEXT3 


The  development  of  the  programed  text  involved  several  suc¬ 
cessive  phases  of  work:  the  selection  of  course  objectives  and 
content;  the  selection  of  training  strategies;  the  drafting  of 
the  programed  materials,  and  initial  program  tryouts  and  revi¬ 
sions;  and  the  field  tryout  and  revision.  These  phases  are 
described  below.  In  addition,  this  section  describes  the  final 
program  and  supplementary  materials,  and  evaluates  the  effec¬ 
tiveness  of  the  text. 

2.1  THE  SELECTION  OF  COURSE  OBJECTIVES  AND  CONTENT 

The  primary  objective  of  the  programed  instructional  text  is 
to  enable  trainees  to  write  statements  in  Intermediate  Query 
Language,  Model  II,  that  could  be  used  to  retrieve  information 
needed  to  solve  moderately  difficult  resource  management  problems, 
stated  in  English.  Subcriterion  objectives  that  were  prerequisites 
to  the  primary  objective  included  an  orientation  to  the  473L  System, 
and  knowledge  of  the  organization  of  information  in  the  data  files 
in  computer  storage. 

Before  preparing  the  draft  instructional  program,  it  was 
necessary  to  define  the  specific  content  of  the  materials  that 
would  most  effectively  and  efficiently  reach  the  established 
terminal  objectives.  Then,  appropriate  training  strategies  were 
selected  on  a  topical  basis,  according  to  their  appropriateness 
for  each  topic.  Since  Query  Language  transitioned  from  Model  I 
to  Model  II  during  the  period  of  development,  the  original  outline 
and  training  strategies  were  specific  to  Query  Language,  Model  I; 
as  Query  Language  evolved,  changes  were  made  in  the  program  to 
correspond  to  the  changes  in  content.  Thus,  revisions  during 
program  development  required  not  just  changes  in  training  strategy 
but  also  substantial  changes  in  program  content. 

2.2  THE  SELECTION  OF  TRAINING  STRATEGIES 

Training  strategies  are  based  on  the  interaction  of  three 
factors:  trainee  characteristics,  the  course  objectives,  and 

specific  characteristics  of  the  course  content. 

2.2.1  Trainee  Characteristics.  The  following  profile  describes 
pertinent  characteristics  of  the  population  of  new  trainees  who 
will  use  this  course:  most  trainees  will  have  reached  a  minimum 
educational  level  of  12  years  of  public  schooling;  they  will  have 
received  a  10-hour  indoctrination  course  on  the  473L  System, 


^The  programed  text  for  Query  Language,  Model  II,  is  presented 
in  reference  3« 
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including  a  one-hour  orientation  lecture  on  the  use  of  Query- 
Language;  they  will  have  received  no  formal  training  in  the  use 
of  Query  Language;  they  will  have  already  acquired  limited  pro¬ 
ficiency  in  using  the  integrated  console  in  the  overlay  mode  of 
operation . 

Since  initial  variability  of  potential  trainees  in  terms  of 
their  prior  knowledge  of  Query  Language  is  negligible,  it  did  not 
appear  urgent  for  the  content  of  instruction  to  be  varied  for 
individual  trainees.  That  is,  there  seemed  to  be  no  need  to  use 
special  techniques  such  as  branching  or  gating  to  accommodate  for 
wide  differences  in  previous  knowledge. 

In  writing  the  instructional  materials,  an  attempt  was  made 
to  gear  the  reading  difficulty  to  the  ninth  grade  level  in  order 
to  facilitate  training  efficiency.  This  was  done  by  exerting 
control  over  sentence  length;  checking  the  level  of  Infrequently 
used  words  and,  whenever  possible,  replacing  those  whose  level 
was  unusually  high;  and,  on  a  more  empirical  basis,  by  revising 
those  steps  which  appeared  difficult  in  terms  of  initial  tryout 
data  .4 

2.2.2  The  Course  Objectives.  The  primary  objective  of  this 
course,  as  mentioned  previously,  is  to  develop  proficiency  in 
writing  Query  Language  statements  for  a  representative  set  of 
problem  types.  This  required  the  development  of  verbal,  con¬ 
ceptual  skills.  The  instructional  programing  techniques  selected 
as  most  appropriate  were  the  use  of  small  steps,  careful  se¬ 
quencing,  overt  written  responding  (with  responses  building  in 
complexity  as  more  material  is  learned),  immediate  confirmation, 
and  self -pacing.  The  selection  of  this  methodology  was  based  on 
previous  experience  at  the  Institutes  with  its  application  to 
similar  training  problems. 

2.2.3  Characteristics  of  the  Course  Content.  The  selection  of 
an  overall  programing  methodology  was  primarily  dependent  on 
trainee  characteristics  and  the  terminal  objectives.  More  spe¬ 
cific  training  strategies  were  selected  on  the  basis  of  the 
characteristics  of  the  subject  matter. 

One  of  the  strategies  contingent  on  the  course  content  was 
the  extent  to  which  the  data  in  computer  storage  should  be  sampled 
by  the  training  problems.  The  stored  information  in  System  473L 
is  extensive  and  AF  documents  are  available  to  aid  the  operator 
in  determining  the  location  and  format  of  the  data  to  be  accessed  . 
Further,  some  of  the  data  files  have  varying  frequency  of  use;  an 
individual  file  might  be  used  rarely  by  some  Air  Force  personnel 
and  very  frequently  by  others,  depending  on  the  primary  function 


%*he  grade  level  of  words  was  determined  by  referencing  Thorndike, 
E.  L.  &  Lorge,  I.  The  teacher's  word  book  of  30*000  words . 

New  York:  Columbia  University,  1959- 
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of  the  particular  user  group  (e.g.,  logistics,  personnel,  etc.). 
Therefore,  it  was  decided  that  while  basic  training  should  exhaust 
the  possible  capabilities  and  variations  in  format  of  Intermediate 
Query  Language,  this  training  could  not  feasibly  be  applied  to  an 
exhaustive  sample  of  the  actual  data  in  computer  storage.  There¬ 
fore,  the  range  of  data  for  which  the  trainees  were  to  write 
Query  Language  statements  during  the  training  course  was  not 
intended  to  be  exhaustive  but  to  constitute  a  representative 
sample  of  the  types  of  information  in  computer  storage.  This 
approach  was  expected  to  minimize  training  time  by  reducing  the 
redundancy  Inherent  in  an  approach  that  sampled  all  data.  To 
provide  for  effective  transfer  of  training  for  the  retrieval  of 
types  of  data  not  specifically  covered  in  the  training  course, 
the  trainees  were  to  receive  extensive  practice  in  the  use  of 
the  AF  documents  specifying  the  location  and  format  of  data. 

The  sequencing  of  training  steps  was  dependent  on  the 
intrinsic  interaction  among  the  items  to  be  covered.  For  example, 
understanding  the  structure  of  a  Query  Language  statement  requires 
knowledge  of  the  organization  of  Information  in  computer  storage. 
Therefore,  the  structure  of  the  computer  files  was  taught  early  in 
the  course.  As  in  this  case,  whenever  possible  sequencing  was 
designed  to  optimize  the  amount  of  positive  transfer  from  one  con¬ 
cept  to  another. 

2.3  DESCRIPTION  OF  THE  FINAL  PROGRAM 

2.3.1  The  Preprogram .  A  preprogram  of  34  frames  was  oeveloped  to 
familiarize  the  trainees  with  the  characteristics  of  the  textual 
program  and  take  the  place  of  an  initial  orientation  lecture  on 
self -instructional  programing.  This  preprogram  is  designed  to  be 
taken  prior  to  the  programed  books  on  Query  Language  and  would 
enable  trainees  to  start  the  Query  Language  textual  program  with¬ 
out  any  additional  assistance.  Thus,  the  preprogram  reduces  to  a 
bare  minimum  the  need  to  utilize  the  time  of  training  personnel. 

2.3.2  The  Textual  Program  on  Query  Language. 

2. 3. 2.1  Number  of  frames.  Appendix  A  shows  the  number  of  frames 
in  each  volume  of  the  programed  text  on  Query  Language.  The  total 
number  of  frames,  counting  the  preprogram,  is  1448. 

2. 3. 2. 2  Content  of  frames.  The  program  is  cumulative  in  that  the 
ability  to  write  Query  Language  statements  depends  on  a  number  of 
subcriterion  skills,  and,  therefore,  the  subcriterion  skills  were 
developed  first. 

The  dependence  of  the  criterion  upon  subcriterion  skills  is 
reflected  by  the  sequence  of  topics  in  the  program.  Thus,  the 
initial  sections  of  the  program  develop  the  subcriterion  skills: 
successively,  133  frames  are  devoted  to  an  orientation  to  the 
473L  System;  43  frames  cover  the  organization  of  information  in 
the  data  files;  21  frames  cover  the  selection  of  the  file  from 
which  the  desired  data  might  be  obtained;  and  51  fran.es  cover  the 
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selection  of  the  attributes  and  values  that  need  to  be  specified; 
the  major  portion  of  the  program,  consisting  of  977  frames,  pro¬ 
vides  intensive  development  of  the  ability  to  write  Query  Language 
statements.  The  last  three  volumes,  consisting  of  189  frames, 
give  extensive  guidance  and  practice  in  the  use  of  the  Air  Force 
Data  Control  Manuals,  so  that  the  trainee  can  learn  to  write 
"complex”  Query  Language  statements,  which  require  great  facility 
in  working  with  a  number  of  data  files,  and  so  that  the  trainee 
can  become  proficient  in  using  these  manuals  and  be  able,  in  the 
future,  to  readily  access  data  from  files  that  he  has  not  pre¬ 
viously  worked  with  or  from  files  that  he  is  not  extensively 
familiar  with. 

The  strategy  used  in  developing  proficiency  in  writing  Query 
Language  statements  is  to  require  that  trainees  learn  the  func¬ 
tional  correspondence  between  the  rules  of  format  and  the  appro¬ 
priate  computer  operations.  An  important  factor  in  this  develop¬ 
ment  of  proficiency  is  the  ability  to  understand  the  organization 
of  the  data  files.  The  importance  of  this  factor  is  illustrated 
by  the  frames  below,  excerpted  from  Volumes  XI  and  XII.  For 
illustrative  purposes,  the  frame  answers  are  also  shown  here, 
enclosed  in  [  ]  and/or  underlined  where  appropriate. 

Volume  XI:  MIN  and  MAX  Functions 

MIN  and  MAX  as  Values 

1.  Another  function  which  Query  Language  can  perform  is  to  select 
entries  with  maximum  and  minimum  values  of  a  particular  attribute. 
To  find  an  airfield  with  the  longest  runway,  you  would  want  to 
select  the  entry  with  the  (minimum/maximum)  |~ maximum ]  value  for 
RNWY  LENGTH.  L  J 


3.  Which  of  the  two  entries  below  will  be  selected? 
(Airfield  name)  |~Hobson  j 

. . .  WITH  RNWY  LENGTH  =  MAX  . . . 

AFLD  NAME  HOBSON  CHECKER 

RNWY  LENGTH  1500/  11500 

RNWY  WIDTH  150  200 


5.  When  MAX  or  MIN  are  used,  the  entry  must  meet  aLl  other 
qualifications  first.  In  the  following  example,  Hobson  has  the 
longest  runway.  However,  it  does  not  qualify  because  it  is  not 
in  the  ^United  Statesj .  The  entry  that  will  be  selected  is  fPow 

. . .  WITH  COUNTRY  =  USA ,  RNWY  LENGTH  =  MAX  . . . 


AFLD  NAME 

HOBSON 

DOBSON 

HOWE 

DOW 

COUNTRY 

CANADA 

USA 

FRANCE 

USA 

RNWY  LENGTH 

13700 

9500 

9000 

10000 

10 


14 .  If  there  is  a  tie  for  the  minimum  or  maximum  value,  you  can 
select  between  those  entries  by  specifying  a  further  minimum  or 
maximum  modifier.  A  second  modifier  will  be  in  effect  only  if 
there  is  a  tie  resulting  from  the  first  modifier.  In  the  fol¬ 
lowing  example,  circle  the  entries  which  will  be  selected  by  the 
first  modifier.  Of  the  circled  entries,  which  one  will  be  selected 
by  the  second  modifier?  Place  a  check  above  that  entry. 


. . .  WITH  RNWY  LENGTH  =  MAX, 


AFLD  NAME  ABLE 

RNWY  LENGTH  10000 

RNWY  WIDTH  150 


BAKER 

12000 

200 


RNWY  .WIDTH  =  MIN 


/CHARLIE  \ 
12500 
\  150  ) 


DOBBS 

10000 

250 


MIN  and  MAX  as  Computed  Attributes 


22. 


WIT 
UNIT  LOC 
MDS 

ACFT  RDY 
CRWS  RDY 


131FIS 

AMARILLO 

F100F 

19 

16 


Suppose  you  want  to  know  whether  the  131st  FIS  at  AMARILLO 
has  enough  crews  to  man  the  operationally  ready  aircraft.  You 
can  answer  this  by  determining  which  of  two  attributes  has  a 
minimum  value;  the  two  attributes  to  be  compared  are 
[ACFT.RDY1  and  fCRWS  RDY] . 


either  order 


23- 


UNIT 

UNIT  LOC 
MDS 

ACFT  RDY 
CRWS  RDY 


131FIS 

AMARILLO 

F100F 

19 

16 


The  values  of  ACFT  RDY  and  CRWS  RDY  may  be  compared  within 
each  column,  or  entry.  Therefore,  the  minimum  (or  maximum)  of 
these  values  may  be  determined  for  each  column,  or 


gentry! 


24.  For  a  given  set  of  alternate  attributes,  to  specify  the  one 
with  the  maximum  value,  you  would  use  [MAX]  (attribute,  attribute,  . 
to  specify  the  one  with  th^  minimum  value,  you  would  use  |~ MIN 
(attribute,  attribute,  .  . .  )1.  L 


27.  For  all  tanker  aircraft,  we  want  to  know  the  MDS,  anc  the  A VC 
TKCFF  WT  or  the  AVG  LAND  WT,  whichever  is  lower.  Complete  the  Query 
RETRIEVE  ACFT  CHAR  WITH  ACFT  CAT  =  TANKER  THEN  LIST  MDS, 

I’M  IN  (AVG  TKOFF  WT,  AVG  LAND  WT)'  “| 
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Volume  XII;  GREATEST  and  LEAST 


2.  When  an  attribute  may  have  more  than  one  value  per  entry,  and 
the  values  are  numeric,  we  may  not  want  to  print  out  all  the 
values.  We  may  want  to  print  out  only  the  value  which  Is  greatest 
or,  on  the  opposite  extreme,  the  ["least  (smallest,  etc.)~j. 


5.  When  an  attribute  has  more  than  one  numeric  value  per  entry, 
you  can  write  a  QL  statement  to  find  the  ["greatest  or  fleast"! 
value  for  that  entry.  L  J _  L  J 


8.  In  the  table  shown  below,  for  which  attribute  could  you  find 
the  greatest  or  least  value?  j^STAGE  CRWS  RQ.j 

PLAN  IDENT  file 


PLAN  IDENT 

1234A 

1234B 

1235A 

1235B 

BASIC  PLAN 

123 

123 

123 

123 

MDS 

FI  0OA 

F100A 

B32 

B47E 

F100B 

FI  00C 

F100D 

B58 

STAGE  CRWS  RQ 

20 

31 

16 

21 

15 

10 

14 

25 

33*  While  attribute  =  MAX  Is  designed  to  select  the  one  entry_ 
that  has  the  maximum  value  for  the  specified  attribute,  attribute  = 
GREATEST  selects  one  value  of  the  specified  attribute  for  each 
[entry],  and  a  correspond lngfvalue  ]  for  every  other  attribute  that 
belongs  to  the  same  [  subset]1- . 


37.  We  want  to  know  the  plan  Identifications  and  all  the  sus¬ 
taining  phase  data  available  pertaining  to  the  sustaining  MDS's 
with  the  largest  number  of  total  flying  hours  for  each  entry  In 
the  PLAN  IDENT  file.  Write  the  Query.  Obtain  a  display  In  any 
format . 

RETRIEVE  PLAN  IDENT  WITH  TOTAL  FLY  HRS  s  GREATEST  THEN 

LIST  PLAN  IDENT,  SUSTAIN  MDS,  TOTAL  FLY  HRS,  TOT  ELAPSED 

TIKE! - - - - 


2. 3. 2. 3  Reference  materials .  Aside  from  the  Illustrations  which 
are  presented  on  Individual  frames  and  the  AF  documents  specifying 
the  location  and  format  of  data,  the  textual  program  utilizes  36 
exhibits  external  to  the  programed  books.  Of  these,  33  are  bound 
together  In  one  volume  and  are  used  to  display  diagrams,  models, 
descriptions  of  data  flies,  etc.,  that  are  frequently  referenced^ 
by  the  program  to  avoid  unnecessary  repetition  of  material  on 
individual  frames.  An  additional  exhibit  that  Is  bound  separately 
consists  of  a  series  of  review  panels.  The  major  types  of  exhibits 
and  their  use  are  described  below. 
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a )  Schematic  Diagram  of  Information  Flow  In  the  473L  System . 
This  diagram,  shown  in  Figure  1,  was  prepared  as  an  aid  in  devel¬ 
oping  an  understanding  of  the  473L  System,  in  which  Query  Language 
is  a  major  mode  of  data  retrieval.  This  diagram  shows  the  possible 
flows  of  information  in  the  473L  System  when  Query  Language  is  used 
for  data  retrieval:  it  shows  the  types  of  input  components,  the 
flow  of  input  through  the  computer  to  the  storage  files,  ana  the 
flow  of  output  data  to  the  output  component .  This  aid  is  used 
exclusively  in  the  first  part  of  the  program,  which  presents  a 
brief  orientation  to  the  473L  System. 

b)  Illustrations  of  Various  System  Components .  These  exhibits 
are  intended  to  facilitate  the  trainee~Ts  orientation  to  the  473L 
System . 

c )  A  Description  of  the  Procedural  Steps  Required  to  Retrieve 
Data  for  a  Specified  Problem  Using  Query  Language.  This  exhibit, 
shown  in  Figure  2,  gives  a  brief,  step-by-step  description  of  the 
overall  process  of  problem  solution,  including  an  initial  problem, 
the  operator's  actions,  the  operation  of  the  computer,  anci  the 
output  of  data. 

c )  A  Description  of  the  Data  Files.  This  exhibit  is  used  to 
help  orient  the  trainee  to  the  names  of  the  data  files  anc  their 
general  contents.  An  excerpt  from  this  exhibit  is  given  in 
Figure  3» 

e )  Charts  Showing  the  Conceptual  Categorlzat ion  of  the  Data 
Files .  These  charts  group  the  data  files  according  to  subject 
matter  (e.g.,  airfield  information,  personnel  information,  etc.) 
and  according  to  the  level  of  data  description  (e.g.,  narrative 
descriptions,  detailed  characteristics,  index  oata,  etc.).  These 
are  used  to  help  the  trainee  become  proficient  in  accessing  the 
proper  file . 

f )  Sample  Tables  with  an  Incomplete  Listing  of  Attributes 
and  Typical  Values .  These  tables  are  used  in  the  program  to 
develop  the  trainee's  concept  of  the  organization  of  data  stored 
in  the  data  files.  One  such  table  is  shown  in  Figure  4.  Shorter 
excerpts  from  tables  are  frequently  shown  on  individual  frames. 

g)  An  Example  of  How  Information  is  Retrieved  from  the  Data 
Files  Using  Query  Language .  This  exhibit,  shown  in  Figure 
illustrates  the  overall  retrieval  process  by  presenting:  a  sample 
set  of  files,  including  an  excerpt  (with  hypotnetical  values)  from 
the  file  from  which  information  is  to  be  retrieved  for  the  solu¬ 
tion  of  a  problem;  and,  the  identification  of  each  element  of  a 
Query  Language  statement  with  the  successive  selection  from  these 
computer  storage  files  of  a  particular  file,  particular  columns 

of  data,  and  the  desired  output  data. 

h)  The  Basic  Output  Directors.  This  exhibit  shows  the  basic 
output  directors,  the  output  device  for  each,  and  the  format  of 
output  data. 
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i )  Examples  of  Corrective  Action  Following  Error  Messages .  As 
part  of  the  initial  orientation  to  the  473L  System,  an  exhibit  is 
used  to  illustrate  the  type  of  display  on  the  integrated  console's 
CRT  display  screen  for  several  different  types  of  error,  and  the 
action  that  should  be  taken  in  each  case  to  correct  the  erroneous 
Query  Language  statement. 

j )  Rules,  Formats,  Examples  of  Input  and  Output,  and  Other 
Teaching  Aids  for  Special  Query  Language  Functions  (such  as  GCD, 

SUM,  titles,  sorts,  special  directors  using  the  SAVE  table,  CHECK, 
and  Complex  Queries).  These  exhibits  are  used  to  help  tne  trainee 
learn  and  discriminate  between  the  different  formats  and  functions 
of  each  Query  Language  function.  Examples  of  these  exhibits  are 
Figures  6  and  J. 

k)  Review  Panels.  These  are  intended  to  be  used  for  a  cumula¬ 
tive  review  of  the  first  70^  of  the  program  (through  Volume  XIX) . 
They  summarize  the  information  taught  about  the  sequencing  and 
functions  of  the  basic  Query  Language  elements,  such  as  the  file 
indicator,  the  qualifier,  etc.;  the  organization  of  data  in  the 
data  files;  the  output  devices  and  formats;  and,  the  function, 
format,  rules  for  use,  and  relation  to  the  other  parts  of  a  state¬ 
ment,  of  specific  parts  or  types  of  the  basic  elements,  such  as 
attributes,  special  kinds  of  attributes  (e.g.,  SUM  and  GCD),  values, 
etc . 


Fig.  1.  Schematic  diagram  of  planned  information  flow  in 
the  design  for  a  second -generation  473L  System. 
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Step  1.  We  define  the  problem.  In  this  example,  we  need  to  find 
out  the  names  of  all  airfields  with  runways  of  at  least  10,000  ft. 
This  information  is  stored  somewhere  in  the  3055  data  files.  All 
we  need  to  do  is  get  it  out. 


STORAGE  UNIT 


DATA 

L-3055 

|71  FILES)  ! 

DATA 

BASE 

—I 

PROGRAMS  | 

_ 1 _ 

Millions  of  items 
of  information  -- 
similar  to  a  large 
library  --  revised 
every  day. 


L-3055  COMPUTER 


Step  2.  The  operator  selects  a  program  (RETRIEVE)  and  the  correct 
file  (AIRFIELDS)  from  a  description  of  the  data  base  in  specially 
prepared  AF  documents. 


Step  3.  The  operator  selects  the  correct  attribute  names  ( RNWY 
LENGTH  and  AFLD  NAME)  from  a  description  of  the  data  base. 


Step  4.  The  operator  writes  a  Query  Language  statement. 

RETRIEVE  AIRFIELDS  WITH  RNWY  LENGTH  >  10000 
THEN  LISTV  AFLD  NAME,  RNWY  LENGTH “] 


Step  5-  The  operator  types  in  the  QL  entry  on  the  Integrated  Console. 


TO  COMPUTER 


Fig.  2.  A  description  of  the  proposed  procedural 
to  retrieve  data  for  a  specified  problem  using  Query 


steps  required 
Language . 
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Step  6.  The  computer  selects  the  RETRIEVE  program  and  starts 
processing  the  QL  statement. 


PROGRAMS 
RETRIEVE 


!  PROCESSING 
UNIT 


QL  STATEMENT 
FROM  1C 


L-3055  COMPUTER 


Step  7. 


The  RETRIEVE 
program 
selects  the 
correct 
file. 


AIRFIELDS 


DATA_ _ 

RETRIEVE 
PROGRAM  | 


PROCESSING 

UNIT 


L-3055  COMPUTER 


and  the 
correct 
attribute 


AFLD  NAME 
RNWY  LENGTH 


AIRFIELDS 


and  the 
correct 
qualifying 
columns , 


RNWY  1  13  000 

11,000 

L  L 

—  f — ' 

and  the 
correct 
output 
attributes 
specified  by 
the  selector. 


(  LOGAN  ) 

(  11,000  ) 


] 


Step  8.  The  computer  sends  the  output  to  the  selected  device 
(LISTV  specifies  the  CRT  display  screen  of  the  IC). 


L-3055  COMPUTER 


INTEGRATED  CONSOLE 


Step  Q.  The  desired  data  is  outputted  in  the  desired  format 
(LISTV  specifies  vertical  format). 


Pig .  2  (cont .  ) 

1 6 


File  Name 
Abbreviation 
ACFT  CHAR 


ACFT  MSL  MAINT 


ACFT  PLAN  FAX 


AFLD  CLIM  INDEX 

AFLD  CLIM  REEL 

AIRFIELD  INDEX 

AIRFIELDS 

AIRFIELD  CLIM 

AIRFIELD  REEL 

AIRMEN  AFSC 


File  Name 

Clear _  _ Description _ 

Aircraft  Physical,  operational,  and 

Characteristics  logistic  information  on  USAF 

and  civilian  reserve  aircraft. 


Aircraft/Missile  Scheaules  for  normal  anc  com- 
Maintenance  pressed  rates  of  maintenance 

in  a  depot  for  each  MDS,  ana 
the  origin  and  destination 
unit  of  the  aircraft. 


Aircraft 

Planning 

Factors 


Factors  used  in  solving  opera¬ 
tional  problems  dealing  with 
aircraft/airfield  compatibility 
and  traffic  compatibility,  and 
the  planning  for  contingencies 
that  involve  aircraft  anc-  war 
materiel  items. 


Airfield  Location  o^  current  and  backup 

Climatology  tape  reels,  and  the  reel  number 

Reel  Index  File  in  which  data  for  a  given 

country  is  located  . 


Airfield 
Climatology 
Reel  File 


Climatic  data  such  as  tempera¬ 
ture,  precipitation,  and  fre¬ 
quency  of  occurrence  of  speci¬ 
fic  ceiling  and  visibility 
conditions  by  month  for  approxi¬ 
mately  3500  airfields. 


Airfield  Tape  Location  of  current  anc  backup  tape 

Reel  Index  File  reels,  and  the  reel  number  in 

which  data  for  a  given  country 
is  located  . 


Airfield  Disk  Basic,  frequently  used  cata 

File  concerning  airfields  (rnwy 

data  only  for  longest  runway). 


Airfield 

Climatology 

File 


Monthly  meteorological  cata 
on  airfields  in  the  disk  and 
reel  airfield  files. 


Airfield  Reel  All  data  concerning  airfields. 

File  including  climatology  data. 


Airmen  AFSC  (Air  A  breakdown  of  the  AFSC's, 
Force  Specialty  with  the  meanings  of  each 
Code)  File  part  of  the  code. 


Fig.  3.  A  partial  listing  and  description  of  the  data  files. 
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AIRFIELD  REEL* 


AFLD  NAME 

COUNTRY 

STATE 

AVGAS  CAP** 
RNWY  LENGTH  1 
2 

3 

4 

RNWY  WIDTH  1 
2 

3 

4 

RNWY  SFC  1 
2 

3 

4 

NAVA  ID 
FACILITY 


ANDREWS 

USA 

MD 

200 

12000 

11000 

9000 

6000 

230 

200 

190 

150 

CONCRT 

CONCRT 

ASPHALT 

ASPHALT 

TACAN 

AIR/GND 

TOWER 


LOGAN 

USA 

MASS 

150 

10000 

9500 

8000 

210 

200 

180 

CONCRT 

CONCRT 

CONCRT 

TACAN 

GCA 

TOWER 


MACD ILL 
USA 
FLA 
180 
9000 
8050 
7500 

190 

210 

180 

ASPHALT 

CONCRT 

ASPHALT 

ILS 

TOWER 


*One  difference  between  the  AIRFIELD  REEL  file  and  the  AIRFIELDS 
file  is  that  the  AIRFIELDS  file  is  on  disk  and  contains  runway 
data  for  only  the  longest  runway,  but  the  AIRFIELD  REEL  file  is 
on  tape  and  contains  data  for  all  runways. 

**Values  for  AVGAS  CAP  are  in  thousands  of  gallons  (K  GAL). 


Fig.  4.  An  incomplete  table  with  typical  values. 
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Fig.  5-  An  example  of  how  Information  is  to  be  retrieved 
from  the  data  files  using  Query  Language. 

19 


format 


functions 


a.  simple  form  --  one  SUM  control  Will  sum  values  of 

attribute  and  one  summed  summed  attribute  for 

attribute.  each  value  of  the  SUM 

...  SUM  BY  SUM  control  attribute  control  attribute, 

(summed  attribute)  ... 

Example  . . .  SUM  BY  UNIT  (ACFT  POSS ) . . . 


b.  general  form  --  multiple  SUM  control 
attributes  and  summed  attributes. 

. . .  SUM  BY  SUM  control  attribute  EY 
SUM  control  attribute  BY  SUM  control 
attribute  (summed  attribute,  summed 
attribute,  . . .  )  ... 

Example  . . .  SUM  BY  UNIT  BY  MDS  (ACFT 
RDY,  CRWS  RDY )  ... 


Will  sum  values  of  each 
summed  attribute  for 
every  combination  of 
values  of  the  SUM  con¬ 
trol  attributes. 


c.  short  form  (no  SUM  function  in  the  Will  provide  a  total  sum 
qualifier).  (over  all  entries  that 

...  SUM  (summed  attribute,  summed  qualify), 

attribute,  . . . )  ... 

Example  . . .  SUM  (ACFT  RDY,  ACFT 
POSS )  . . . 


d.  short  form  if  a  SUM  function  has 
been  specified  in  the  qualifier. 
...  SUM  ... 

. . .  LISTH  UNIT,  UNIT  LOC, 


Outputs  sums  already 
computed  In  the  qualifier. 


Fig.  6.  Formats  and  functions  of  SUM  in  the  output  selector. 
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entry- 


entry 


POB  (C) 

ITEM  (C) 

• 

• 

(S) 

STORAGE  LOC  ' 

PARKERSBURG 

CHAFF  RR131 

• 

m 

SAMPSON 

CHAFF  RR131 

• 

• 

I 

ARNOLD 

MULDOON 

VICTORY 

• 

ARNOLD 

VICTORY 

• 

• 

• 

SL  ASSET 

3000 

• 

7000 

• 

3000 

4000 

• 

7000 

1.  RETRIEVE  MATERIEL  STATUS  WITH  ITEM  =  CHAFF  RR131,  STORAGE 
LOC  =  VICTORY  THEN  LIST  SL  ASSET  "1 


output : 


SL  ASSET 


2.  RETRIEVE  MATERIEL  STATUS  WITH  ITEM  =  CHAFF  RR131,  STORAGE 
LOC  =  VICTORY  THEN  LIST  SUM  (SL  ASSET)' 


output : 


SUM  SL  ASSET 


3- 


RETRIEVE  MATERIEL  STATUS  WITH  ITEM  =  CHAFF  RR131  THEN  LIST 
SUM  BY  STORAGE  LOC  (SL  ASSET)-) 


output : 


STORAGE  LOC 
SUM  SL  ASSET 


Fig.  7*  Examples  of  input  and  output  formats  for  SUM. 
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2. 3- 2. 4  Program  format .  Each  "frame"  consists  of  two  parts,  a 
stimulus  panel  and  an  answer  panel.  A  stimulus  panel  consists  of 
written  material,  including  at  least  one  incomplete  statement; 
missing  words  or  phrases  are  indicated  by  blanks.  Answer  panels 
contain  the  missing  words  or  phrases.  In  some  cases,  where  this 
appeared  appropriate,  the  missing  words  or  phrases  are  presented 
in  their  appropriate  context  (from  the  stimulus  panel)  on  the 
answer  panel. 

Stimulus  panels,  and  likewise  response  panels,  are  presented 
three  to  a  page,  on  sequential  pages  of  bound  volumes.  Frames 
that  relate  to  a  relatively  discrete  topic  are  bound  together; 
thus,  separate  topics  are  usually  denoted  by  separate  volumes. 

The  sequence  of  stimulus  and  answer  panels  is  shown  in  the  schematic 
on  the  following  page . 

2.3.3  Computer  Exercises.  Although  developing  proficiency  in 
using  the  integrated  console  is  not  a  primary  task  of  the  pro¬ 
gramed  text,  it  was  suggested  that  some  experience  in  actually 
using  the  console  to  retrieve  data  with  Query  Language  statements 
would  have  a  motivating  affect.  In  addition,  the  exercises  are  to  be 
used  to  permit  trainees  to  become  adept  at  Interpreting  and 
reacting  to  Error  Messages  related  to  Query  Language  inputs,  dis¬ 
played  on  the  CRT  display  screen. 

The  computer  exercises  are  designated  for  use  at  points 
scattered  throughout  the  course.  The  instructions  for  these 
exercises  are  bound  together  In  an  Exercise  Book. 

2.3.4  Test .  Two  tests  were  developed  for  use  in  evaluating  the 
Query  Language  course.  One  was  used  during  the  preliminary  try¬ 
outs  to  test  the  first  70$  of  the  program.  The  other  test  is  the 
final  posttest,  which  is  cumulative  over  the  entire  content  of 
the  program;  a  copy  of  this  test  Is  shown  in  Appendix  B.  A  copy 
of  the  final  test  answer  and  scoring  key  is  in  Appendix  C. 

The  final  posttest  is  designed  primarily  to  evaluate  pro¬ 
ficiency  in  writing  Query  Language  statements;  it  requires  the 
trainee  to  write  a  representative  sample  of  the  types  of  Query 
Language  statements  taught  throughout  the  program.  This  test 
was  used  at  the  completion  of  the  programed  course  to  evaluate 
the  overall  effectiveness  of  the  course,  and  to  indicate  neces¬ 
sary  specific  revisions  in  the  program.  In  the  future,  this 
test  can  be  used  to  evaluate  the  proficiency  of  individual 
trainees.  The  almost  exclusive  emphasis  in  the  final  test  on 
proficiency  in  writing  Query  Language  statements  reflects  both 
the  importance  of  this  objective  and  the  fact  that  the  level  of 
proficiency  in  writing  Query  Language  statements  also  reflects, 
in  most  cases,  the  level  of  proficiency  in  the  subcriterion 
behaviors . 
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2.4  THE  DRAFTING  OF  FRAMES,  INITIAL  PROGRAM  TRYOUTS,  TECHNICAL 
REVIEW,  AND  REVISIONS 

Draft  frames  were  based  on  the  initial  specification  of 
course  objectives,  content,  and  training  strategies.  As  succes¬ 
sive  portions  of  the  program  were  completed,  they  were  given  pre¬ 
liminary  tryouts  and  a  review  for  technical  accuracy,  and  revised 
on  the  basis  of  the  tryout  data  and  technical  comments.  Due  to 
the  evolving  nature  of  Query  Language,  some  changes  in  frames 
were  also  necessitated  by  changes  in  the  language  Itself. 

Prior  to  the  field  tryout  on  the  entire  text,  there  were  two 
preliminary  tryouts  and  revisions.  The  volumes  evaluated  in  the 
first  tryout  constituted  about  38$  of  the  final  text.  The  volumes 
in  the  second  tryout  included  the  volumes  used  in  the  first  tryout 
plus  additional  materials;  the  total  number  of  volumes  in  the 
second  tryout  amounted  to  about  70$  of  the  final  text.  The  two 
preliminary  tryouts  and  revisions  are  described  below. 

2.4.1  Subjects .  All  of  the  preliminary  tryouts  were  conducted 
at  the  Pentagon  with  a  small  number  of  Air  Force  personnel  who 
were  typical  of  the  target  population.  Four  trainees  participated 
in  the  first  tryout.  In  the  second  tryout,  these  trainees  took 
only  the  materials  that  had  been  drafted  subsequent  to  the  first 
tryout;  two  new  trainees  took  all  of  the  materials  that  had  been 
drafted  up  to  this  point  (approximately  70$  of  the  final  program). 

2.4.2  Results .  Since  the  number  of  subjects  for  the  preliminary 
tryouts  was  small,  the  results  were  used  for  the  very  limited 
purpose  of  revising  the  programed  text.  Generalizations  to  other 
programed  materials  are  not  appropriate. 

Percent  correct  pretest  scores  for  individual  trainees  were 
low:  the  median  score  for  the  six  trainees  participating  in 

the  two  preliminary  tryouts  was  2.9$;  the  range  was  0.5$  to  10.0$. 

For  the  preliminary  tryouts,  the  average  error  rate  on  each 
draft  volume  of  the  programed  text  (not  counting  Volume  I  -  the 
preprogram,  for  which  the  error  rate  was  negligible)  is  given  in 
Table  1  below.  The  error  rates  ranged  from  3*85$  to  24.2$.  The 
median  error  rate  for  the  draft  volumes  was  8.5$.  As  shown, 
only  two  out  of  14  volumes  had  an  error  rate  exceeding  10$. 

Volumes  II  through  IX  were  used  in  draft  form  in  the  first 
tryout;  the  average  error  rate  on  this  portion  of  the  program 
was  8.3$.  In  addition  to  the  revised  volumes  II-LX,  Volumes  X 
through  XIV  were  used  in  draft  version  in  the  second  tryout; 
the  average  error  rate  on  draft  volumes  X-XIV  was  7-6$. 
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TABLE  1 


Average  Error  Rates  on  Draft  QL  Volumes  That 
Were  Used  in  the  Preliminary  Tryouts 


First 

Tryout 

Second 

Tryout 

Volume 

Average  Error  Rate 

Volume 

Average  Error  Rate 

II 

6.6$ 

X 

8.8$ 

III 

8.5 

XI 

8.1 

IV 

23.4 

XII 

4.7 

V 

9.1 

XIII 

7.0 

VI 

9.6 

XIV 

8.7 

VII 

3.65 

VIII 

24.2 

IX 

5.2 

Individual  error  scores  on  the  first  70$  of  the  draft  program 
and  percent  correct  posttest  scores  on  the  test  covering  this  por¬ 
tion  of  the  program  are  given  in  Table  2  below  for  each  preliminary 
tryout . 


TABLE  2 

Percent  Error  for  Each  Trainee  on  the  Program  and  Percent 
Correct  on  the  Test  for  Each  of  the  Preliminary  Tryouts 

Tryout  1  S2  S3  S4 

Draft  Program  9*9$  6.0$  9*9$  7.5$ 

Test  96$  95$  90$  79$ 

Tryout  2  S3  Sg 

Draft  Program  2.8$  6.2$ 

Test  92$  88$ 


The  average  posttest  score  for  the  first  tryout  was  90$  and 
likewise  for  the  second  tryout.  In  view  of  the  high  posttest 
scores  and  the  fairly  low  error  rates  on  even  the  draft  volumes  --  with 
a  couple  of  exceptions  --  and  considering  the  apparent  lack  of 
relationship  between  program  error  rates  and  proficiency  on  the 
posttest,  revisions  were  mostly  based  on  the  technical  review  and  the 
critical  comments  of  the  trainees. 
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2.5  FIELD  TRYOUT  AND  REVISION 


2.5.1  Design .  While  the  scale  of  the  field  test  was  restricted 

by  the  limited  availability  of  personnel  representative  of  the 
target  population,  the  scale  was  sufficient  since  the  field  test 
was  used  exclusively  for  very  specific  evaluative  and  diagnostic 
purposes:  it  was  used  to  evaluate  the  effectiveness  of  the  pro¬ 

gramed  course  in  terms  of  the  specified  objectives,  and  to  diag¬ 
nose  areas  of  Query  Language  that  trainees  had  difficulty  in 
learning,  for  which  revisions  were  required  in  the  training  mate¬ 
rials  . 

As  previously  mentioned,  the  primary  objective  of  the  program 
is  to  enable  Air  Staff  personnel  to  write  Query  Language  statements 
for  the  solution  of  moderately  difficult,  specified  problems. 
Evaluation  of  the  program's  effectiveness  in  achieving  this  objec¬ 
tive  was  based  exclusively  on  the  trainees'  proficiency  on  the 
criterion  test,  which  emphasized  ability  to  write  Query  Language 
statements . 

2.5.2  Subjects.  As  shown  in  Table  3  below,  seven  of  the 
eight  trainees  had  taken  the  earlier  programed  course  on  OTC 
Query  Language,  which  was  the  first  operational  version  of  Query 
Language;  five  had  had  prior  training  on  either  Model  I  or 
Model  II  Query  Language;  and,  three  of  the  trainees  had 
actually  participated  in  the  preliminary  tryouts  with  the  draft 
materials  covering  approximately  70%  of  the  program  on  Inter¬ 
mediate  Query  Language,  Model  II. 


TABLE  3 


Background  Data  for  Trainees  Participating  in  the  Field  Tryout 


Trainee 

Previous 
Training 
in  OTC  QL 

Si 

Yes 

s2 

Yes 

s3 

si 

Yes 

Yes 

s^ 

Yes 

Sg 

Yes 

s7 

Yes 

s8 

No 

Previous  Training 
in  either  Model  I 
or  Model  II  QL 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

No 

No 


Participated  in  the 
Preliminary  Tryouts  on 
the  Draft  Program  for 
Model  II  QL 

Yes 

Yes 

Yes 

No 

No 

No 

No 

No 


Since  the  structure  of  Query  Language  changed  greatly  from 
OTC  to  Model  II,  the  previous  training  of  some  subjects  in  OTC 
Query  was  not  seen  as  a  serious  problem;  in  fact,  in  transitioning 
from  OTC  Query  Language  to  Query  Language,  Model  II,  it  is 
problematic  whether  transfer  of  training  was  positive  or  negative. 
In  either  event,  there  was  an  overriding,  pragmatic  justification 
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for  the  selection  of  these  particular  trainees  to  participate  in 
the  field  tryout:  the  tryout  was  utilized  by  the  Air  Force  as 
an  opportunity  to  meet  current  training  needs. 

All  of  the  trainees  completed  the  programed  course  on 
Intermediate  Query  Language,  Model  II,  but  only  five  took 
the  final  test.  These  were  the  trainees  identified  in  Table  3 
as  S2>  Sg,  Sg,  Sy,  and  Sg. 

2.5.3  Administration  of  the  Field  Tryout.  The  tryout  was  admln- 
istered  by  Air  Force  personnel.  Aside  from  an  initial  briefing, 
direct  monitoring  was  negligible.  Work  on  the  course  was  done  at 
the  Pentagon,  on  a  flexible  schedule.  Other  Air  Force  duties 
were  interpolated,  as  required,  by  each  individual.  Interspersed 
activities  consisted  of  the  trainees'  normal  duties  at  the 
Pentagon  in  their  regular  section.  Trainees  worked  on  the  course 
an  average  of  6-1/2  hours  per  day.  Total  training  time  varied 
from  a  minimum  of  five  days  to  a  maximum  of  eight  days.  Since 
there  was  no  significant  relationship  between  program  error  rates 
and  test  proficiency  in  the  preliminary  tryouts  of  the  program, 
and  since  the  collection  and  analysis  of  program  error  data  is 
very  time-consuming,  it  was  not  considered  worthwhile  (and  it 
was  definitely  not  expedient)  to  collect  program  error  data. 

The  only  data  collected  from  the  field  tryout  were  program  com¬ 
pletion  times  and  percent  correct  scores  on  the  final  test. 

After  the  instructional  program  was  completed,  each  trainee 
was  given  a  posttest  designed  to  evaluate  his  proficiency  in 
writing  Query  Language  statements.  As  mentioned  previously,  a 
copy  of  this  test  appears  in  Appendix  E. 

2.5.4  Results  of  the  Field  Tryout. 

2. 5. 4.1  Completion  times  for  the  programed  materials.  Since 
interruption  of  pi  ogress  on  the  program  to  perform  other  duties 
was  Intermittent,  recording  exact  times  would  have  been  burden¬ 
some.  Therefore,  the  average  number  of  hours  per  day  was  esti¬ 
mated,  at  6-1/2  hours,  and  the  completion  time  for  each  trainee 
was  recorded  in  terms  of  the  total  number  of  days  required  to 
finish  the  program.  The  completion  time  for  each  individual 
trainee  is  shown  in  Table  4  below.  The  average  number  of  days 
for  completion  was  6.3  days;  at  6-1/2  hours  per  day,  this  meant 
an  average  of  roughly  41  hours  to  complete  the  program. 


TABLE  4 


Program  Completion  Time  for  Each  Trainee 


Trainee 


Time 


Trainee 


Time 


6-1/2  days  Sg 

6  days  Sg 

7  days  Sy 

5  days  Sg 


7  days 

5  days 

6  days 

8  days 
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2. 5. 2  Scores  on  the  final  postteat.  Aa  mentioned  previously, 
the  final  test  was  taken  by  only  f'ive  of  the  trainees  partici¬ 
pating  in  the  field  tryout.  Scores  for  individual  trainees  on  the 
final  test  are  shown  in  Table  5. 


TABLE  5 

Percent  Correct  Scores  on  Final  Test  for  Individual 
Trainees  Participating  in  the  Field  Tryout 

94.6$ 

46.5 
80.2 
86.8 
72.4 


S2 

f3 

|5 

s7 

s8 


The  average  percent  correct  score  on  the  final  test  was  76.1$. 
The  median  was  80$,  with  scores  ranging  from  46.5$  to  94.6$.  Thus, 
scores  on  the  field  tryout  final  test  were  lower  than  those  obtained 
on  the  posttest  covering  70$  of  the  program,  which  was  used  in  the 
preliminary  tryouts.  The  most  probable  explanation  for  this  is 
twofold:  l)  the  last  30$  of  the  program  had  not  previously  been 

tried  out;  and,  2)  the  last  30$  of  the  program  was  intrinsically 
more  difficult,  overall,  than  the  first  70$,  and  the  final  test 
was  correspondingly  more  difficult  than  the  posttest  used  in  the 
preliminary  tryouts5. 

2.5.5  Revisions  After  the  Field  Tryout.  Since  time  was  a  critical 
factor  in  completing  the  revisions,  revisions  were  restricted  to 
those  based  on  data  from  the  final  test. 

The  first  step  in  revising  the  program  was  to  make  a  few 
additional  technical  revisions  based  on  information  obtained  from 
Air  Force  experts  on  Query  Language.  Further  revisions  in  the 
program  were  based  on  errors  made  on  the  final  test.  For  each 
item  missed  on  the  test,  corresponding  topical  sequences  were 
identified.  Appropriate  revisions  were  made  in  these  sequences. 

The  revision  based  on  test  errors  was  considered  of  critical 
importance  and  ample  time  was  devoted  to  this  work  in  order  to 
adequately  revise  for  all  recurrent  test  errors. 


5as"  seen  in  Table  3>  trainees  S2  and  had  previously  participated 
in  the  preliminary  tryouts  of  the  draft  program;  their  scores  on 
the  test  covering  the  first  70$  of  the  entire  program  were  90$ 
and  79$  >  respectively.  The  substantial  drop  in  SVs  score  would 
seem  to  indicate  that  the  final  test  was  substantially  more 
difficult  (for  him)  than  the  posttest  used  in  the  preliminary 
tryouts . 
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2.6  DISCUSSION  AND  RECOMMENDATIONS 


Generally,  the  results  of  the  field  tryout  Indicated  that  the 
programed  course  can  train  to  a  satisfactory  level  of  proficiency 
over  a  period  of  about  six  days,  using  a  massed  training  schedule 
of  roughly  6-1/2  hours  per  day.  Past  experience  with  a  similar 
programed  course  on  OTC  Query  Language  Indicated  that  a  training 
schedule  requiring  less  time  per  day  achieves  a  slightly  higher 
level  of  proficiency6.  Thus,  If  the  Immediate  need  for  personnel 
trained  in  Query  Language  Is  not  especially  urgent,  a  spaced 
schedule  of  about  two  to  four  hours  per  day  would  be  most  desir¬ 
able.  But  If  Immediate  training  Is  critical,  a  massed  schedule 
could  be  used  with  only  a  small  penalty  expected  in  terms  of  the 
final  proficiency  level. 

For  the  implementation  of  this  course  as  a  training  device, 
it  Is  recommended  that  the  final  test  be  retained  for  evaluative 
and  motivational  purposes.  It  Is  also  recommended  that  the  inte¬ 
grated  console  exercises  be  used  to  demonstrate  data  retrieval 
for  a  variety  of  Queries.  While  these  demonstration  exercises 
would  be  on  the  computer,  they  would  not  be  computer  directed. 

If  it  is  possible  to  provide  the  adaptive,  computer  directed 
training  course  described  in  the  next  section,  this  would  be 
recommended  In  preference  to  the  programed  text. 


A  massed  practice  group  (approximately  seven  hours  per  day)  scored 
84/  on  the  posttest  and  had  an  average  error  rate  of  7.8/.  A 
spaced  practice  group  (approximately  two  hours  per  day)  scored 
90/  on  the  posttest  and  had  an  average  error  rate  of  5.0/.  More 
detailed  data  are  available  In  reference  1. 
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Section  III 


DEVELOPMENT  OF  THE  OPERATIONAL  SPECIFICATION 
FOR  COMPUTER  DIRECTED  TRAINING 


3 . 1  INTRODUCTION 

The  rationale  for  using  computer  directed  training  in  Query- 
Language  was  described  in  Section  1.4.2  of  this  report.  The 
terminal  product  of  this  phase  of  the  project  is  the  Operational 
Specification  for  Computer  Directed  Training  in  Intermediate 

Query  Language,  Model  II,  for  System  473L,  February  lS,t>67~.  This 

operational  specification  completely  describes  the  training  de¬ 
sign  and  the  necessary  operating  procedures  for  the  proposed 
computer  directed  training  capability.  A  description  of  the 
scope  of  programming  required  for  the  implementation  of  this 
capability,  and  the  Impact  it  will  have  on  System  473L  opera¬ 
tional  capabilities  is  contained  in  a  report  prepared  by  the 
Federal  Systems  Division  of  International  Business  Machines,  Inc.; 
this  report  is  entitled  Computer  Directed  Training:  System  473L 
Query  Language,  April  1966B.  The  general  conclusion  stated  in 
this  document,  as  a  result  of  programming  analysis,  is  that  the 
computer  directed  training  capability  would  be  compatible  with 
System  473L  equipment  and  programming  subsystems,  can  be  utilized 
in  a  manner  similar  to  that  of  existing  operational  capabilities 
using  an  overlay,  would  have  little  impact  on  storage  require¬ 
ments  for  data  and  programs,  and  would  nave  no  impact  on  the 
simultaneous  utilization  of  existing  operational  capabilities. 

The  training  design  and  operating  procedures  were  developed 
with  specific  reference  to  the  Llbrascope  3055  computer  and 
Intermediate  Query  Language,  Model  II,  as  they  are  intended  for 
use  in  System  473L.  However,  the  specific  design  described  by 
the  operational  specification  may  be  used  as  a  model  for  the 
development  of  similar  training  programs  in  other  command  and 
control  systems. 

3.2  DEVELOPMENT  OF  THE  OPERATIONAL  SPECIFICATION 

Each  capability  of  System  473L  is  described  in  detail  by  an 
operational  specification.  The  general  contents  and  format  of  a 
System  473L  operational  specification  are  predefined.  To  ensure 
that  the  operational  specification  for  the  proposed  training 
capability  would  meet  approved  473L  standards,  and  to  provide 
guidance  regarding  the  feasibility  of  proposed  training  features, 
the  developmental  process  required  close  interaction  between  the 
A-I-R  project  staff  and  technical  experts  on  System  473L  from  the 
Federal  Systems  Division  of  International  Business  Machines,  Inc. 

^This  document  is  reference  4. 

"’This  document  is  reference  5 • 
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The  operational  specification  evolved  through  predesignated 
stages,  which  required  the  development,  revision,  and  integra¬ 
tion  of  three  successive  parts:  the  training  design,  or  training 
sequence  logic;  the  procedural  flow  diagram;  and  the  specifica¬ 
tion  of  operating  procedures. 

3.3  DESCRIPTION  OF  THE  OPERATIONAL  SPECIFICATION 

The  following  sections  briefly  describe  the  training  design 
and  the  operating  procedures  for  the  computer  directed  training 
capability.  More  detailed  Information  is  available  from  reference 
4. 


3.3.1  The  Training  Sequence  Logic 

3. 3. 1.1  Complexity  of  the  training  design.  An  ideal  design  for 
computer  directed  training  would  permit  maximum  interaction 
between  the  computer  and  the  Individual  trainee,  including  com¬ 
plete  analysis  of  responses  and  computer  generation  of  all  evalua¬ 
tions,  directions,  problems  for  individual  trainees,  and  responses 
to  student-generated  requests,  with  all  communications  in  more  or 
less  unrestricted  English.  However,  this  is  not  feasible  for  use 
with  an  operational  system.  In  developing  the  training  sequence 
logic,  there  was  a  need  to  limit  the  complexity  of  the  training 
design  in  order  to  optimize  the  feasibility  of  implementation  in 
terms  of  a)  the  need  to  minimize  any  possible  conflict  with 
other  System  473L  operational  capabilities;  b)  the  cost  of  de¬ 
veloping  a  computer  program  to  Implement  the  proposed  logic;  and, 
c)  the  cost  of  training  itself  --  this  would  also  Increase  with 
increasing  complexity  of  the  training  design. 

The  major  training  restrictions  imposed  to  increase  the 
feasibility  of  computer  implementation  and  reduce  cost  were: 

1)  computer  analysis  of  trainee  responses  on  designated 
parts  of  QL  statements,  as  opposed  to  analysis  of  com¬ 
plete  QL  statements . 

2)  the  use  of  "canned"  answers  (vs  computer-generated 
answers)  to  problems  requiring  computer  analysis,  so 
that  analysis  of  a  trainee's  response  could  be  accom¬ 
plished  by  a  simple  matching  process. 

3)  the  use  of  fixed  formats  for  training  materials  and 
problems,  so  that  content  --  but  not  format  --  changes 
could  be  made  without  a  programming  change. 

4)  the  use  of  only  one  major  level  of  remedial  training 
after  each  evaluative  problem  section.  Thus,  no 
remedial  loop  is  used  to  correct  errors  on  a  remedial 
sequence  itself. 
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5)  the  use  of  a  limited  number  of  evaluation  ratings  to 
evaluate  a  trainee’s  performance  and  assign  remedial 
materials.  However,  the  error  criteria  for  the 
various  ratings  can  be  stored  in  such  a  way  that 
changes  in  these  criteria  would  be  relatively  easy 
to  make  if  experience  so  dictates. 

3. 3-1*2  Provision  for  adaptation  of  the  capability.  Several 
provisions  were  made  to  permit  adaptation  of  the  capability  for 
anticipated  future  changes  in  the  training  design  and  Query 
Language,  and  to  permit  training  in  an  entirely  different  sub¬ 
ject.  To  achieve  maximum  flexibility  without  any  programming 
change,  the  provisions  for  adaptation  are  through  data  mainte¬ 
nance.  Some  of  the  major  areas  in  which  data  maintenance 
changes  may  be  made  are:  the  content  of  any  cue  --  i.e.,  the 
content  for  any  unit  of  presentation  to  the  trainee;  the  number 
of  cues  within  a  set;  the  criteria  by  which  trainee  errors  are 
evaluated;  and,  the  remedial  continuations  for  trainee  errors. 

3.3 .1.3  The  training  sequence  logic.  The  proposed  computer 
directed  training  capability  has  three  basic  uses  for  training 
in  Query  Language,  Model  II:  l)  to  train  personnel  in  the  use 
of  Intermediate  Query  Language,  Model  II;  2)  to  provide  perform¬ 
ance  data  for  the  trainees  and  supervisory  personnel;  and,  3)  to 
provide  proficiency  maintenance  training  for  trainees  who  have 
previously  completed  this  or  some  other  training  program  in 
Query  Language,  Model  II. 

Salient  features  of  the  training  sequence  logic,  designed 
to  optimize  training  effectiveness  and  efficiency  within  the 
bounds  of  computer  feasibility  and  cost,  include  the  following: 
presentation  of  training  materials  on  the  console  display  screen; 
computer  evaluation  of  trainee  performance  based  on  the  number 
and  kind  of  errors  and  pre-stored  evaluation  ratings;  computer 
determination  of  training  areas  requiring  remecial  work  for 
each  trainee;  computer  determination  of  the  specific  remedial 
sequence  in  each  area  that  is  appropriate  to  the  kind  and  number 
of  trainee  errors;  and,  periodic  provision  of  optional  remedial 
work  for  trainee  selection. 

The  training  course  designed  for  initial  use  has  a  ouilt-in 
research  design  on  the  relative  effectiveness  of  forcing  trainees 
to  take  remedial  work  appropriate  to  the  number  and  kind  of  their 
errors,  in  addition  to  periodic  provision  of  optional  remedial 
work  (called  Research  Group  "A"),  as  opposed  to  the  provision  of 
optional  remedial  work  only  (called  Research  Group  "B'* ).  This 
comparison  would  provide  some  of  the  information  needed  to  make 
revisions  in  the  training  logic  that  would  improve  the  overall 
effectiveness  and  efficiency  of  the  computer  directed  training 
course . 

Except  for  the  remedial  materials,  whose  presentation  is 
contingent  on  trainee  errors  and/or  selection,  the  sequence  of 
training  materials  is  fixed.  The  initial  training  sets  provide 
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a  basic  orientation  to  the  use  of  the  instructional  materials,  the 
use  of  the  console,  the  data  files  accessed  by  Query  Language,  the 
overall  structure  of  Query  Language  statements,  and  the  use  of  the 
Air  Force  Manuals  that  describe  the  general  nature  and  contents  of 
each  data  base  file  and  the  vocabulary  used  to  reference  the  file 
contents.  Subsequent  sets  provide  extensive  and  cumulative 
training  and  evaluation  on  the  use  of  Query  Language  to  retrieve 
desired  information  from  the  data  files.  The  Query  Language 
topics  covered  by  the  computer  directed  training  course  are  de¬ 
signed  to  progress  in  the  order  specified  by  Appendix  D  of  this 
report . 

Any  deviation  from  the  main  training  path  for  the  computer 
directed  training  course  is  contingent  on  the  trainee's  perform¬ 
ance  and/or  his  own  optional  selection  of  remedial  materials. 

The  main  training  path  and  the  remedial  branches  are  shown  in 
the  Training  Sequence  Logic  Flow  Diagram,  Figure  8.  Explanatory 
notes  for  this  logic  on  a  general  level  are  given  in  the  para¬ 
graphs  be low. ^ 

The  Training  Sequence  Logic  Flow  Diagram  shows  the  main 
training  sequence  for  all  cues  (individual  units  of  presentation 
to  the  operator),  points  requiring  a  decision  by  the  operator, 
points  requiring  computer  evaluation  of  a  trainee's  performance, 
points  requiring  computer  determination  of  any  necessary  branch¬ 
ing,  all  remedial  trailing  paths,  and  points  at  which  data  print¬ 
outs  occur. 

To  facilitate  understanding  of  the  Training  Sequence  Logic 
Flow  Diagram,  a  condensed  outline  of  the  basic  training  sequence 
and  remedial  branches  is  presented  below. 


Basic 

X  SETS  (5  sets) 

Y  SETS  (20  sets) 

Training 

LX-PX  --  Eval 

LY- PY  Eva 1-PEY* -Free  Choice  Review 

Sequence 

V" 

i 

*PEY  not  avail  after  Posttest  PY 

Remedial 

Branches 

for 

Excess 

Errors 

* 

A  FIXED  SEQ  FROM 
LAST  LX  (ONE  SEQ, 
REGARDLESS  OF  TYPE 
OF  ERRORS) 

1 

ONE  OR  MORE  REMEDIAL  SEQUENCES 
APPROPRIATE  TO  l)  THE  PARTICULAR 
AREAS  IN  WHICH  ERRORS  WERE  MADE 
AND  2)  THE  NUMBER  OF  ERRORS  MADE 

?For  a  complete  explanation  of  all  points  on  this  diagram,  the 
reader  is  referred  to  reference  4,  the  Operational  Specification 
for  Computer  Directed  Training  in  Intermediate  Query  Language, 

Model  II,  for  System  473L. 
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NOTES:  The  cues  used  by  this  capability  may  be  divided  into  two 
major  classes:  (i)  the  cues  that  provide  a  basic  foundation  for 
the  course  but  do  not  teach  Query  Language ;  these  cues  are  divided 
into  sets  called  MXM  SETS;  and  (2)  the  subsequent  cues  that  cover 
all  aspects  of  intermediate  QL;  these  cues  are  divided  into  sets 
called  "y"  SETS. 

1)  The  X  SETS  COVER  BASIC,  NON-QL  MATERIALS: 
aj  the  use  of  the  Instructional  materials 

b)  the  use  of  the  console  and  the  CDT  overlay 

c)  a  basic  Introduction  to  the  files  and  structure  of  QL 

d)  the  use  of  the  manuals  covering  the  QL  data  base  and 
system  vocabulary 

2)  The  Y  SETS  cover  all  training  and  evaluation  materials 
provided  on  the  use  of  Query  Language  --  i.e.,  all  mate¬ 
rials  not  covered  by  X  SETS 

3)  For  any  X  SET: 

LX  =  training  sequence  --  a  section  within  the  X  SET 
PX  =■  series  of  problems  on  the  LX  --  a  section  within 
the  X  SET 

Eval  —  evaluation  of  errors  on  the  PX  section 

4)  For  any  Y  SET: 

LY  =  training  sequence  --  a  section  within  the  Y  SET 
PY  =  series  of  problems  on  one  or  more  of  the  preceding 
LY's  --  a  section  within  the  Y  SET 
Eval  =  evaluation  of  errors  on  each  area  of  QL  subjected 
to  analysis  by  the  last  PY  section 
PEY  =  series  of  free-form  practice  exercises  (which  the 
trainee  answers  with  complete  QL  statements  and  on 
which  he  receives  feedback).  This  Is  a  section 
within  each  of  the  Y  SETS  except  the  last  --  this 
is  not  available  after  the  posttest 
Free-Choice  Review  =  at  end  of  each  Y  SET,  the  trainee 
has  the  option  of  taking  review  on  any  area  sub¬ 
jected  to  analysis  by  the  last  PY  section 

5)  Throughout  the  entire  program  there  are  four  general 
types  of  sets  --  the  basic,  non-QL  X  SETS  and  three 
kinds  of  QL  Y  SETS.  These  sets  are  described  below: 

a)  5  basic,  non-QL  X  SETS 

b )  13  INDEPENDENT  Y  SETS  for  each  of  which  the  PY  sec¬ 
tion  tests  only  the  materials  covered  by  the  LY  sec¬ 
tion  in  the  same  set 

c)  6  CUMULATIVE  Y  SETS,  used  at  appropriate  points,  in 
which  the  PY  section  tests  all  materials  covered  after 
the  last  CUMULATIVE  Y  SET 

d)  one  POSTTEST  Y  SET:  this  is  the  last  Y  SET,  for  which 
the  LY  section  reviews  all  materials  covered  In  the 
program  and  the  PY  section  is  the  POSTTEST,  which  tests 
all  the  materials  covered  over  all  of  the  Y  SETS  in  the 
program . 
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An  understanding  of  the  Training  Sequence  Logic  Flow  Diagram 
will  also  be  enhanced  by  an  understanding  of  cue  types.  There 
are  two  general  types  of  cues  used  by  this  capability: 

1)  Instructional  cues.  These  cues  are  used  primarily  to 
Instruct  the  operator  (trainee)  in  the  proper  procedure  for 
making  a  transition  from  one  point  in  the  training  sequence  to 
another.  These  instructional  cues  usually  indicate  the  available 
options  for  continuing,  and  in  some  cases  they  provide  feedback 

to  the  trainee  regarding  his  performance  on  the  last  cue  or  series 
of  cues.  On  the  Training  Sequence  Logic  Flow  Diagram,  most  of  the 
instructional  cues  are  represented  by  a  cue  number,  e.g.,  Ql, 
enclosed  in  a  circle,  and  a  brief  description  of  the  cue's  func¬ 
tion.  Instructional  cues  that  provide  an  introduction  to  a 
sequence  of  learning  cues  are  specified  to  the  right  of  the 
bracket  enclosing  the  series.  For  example,  on  page  1  of  Figure  S, 
Q7  is  an  Instructional  cue  that  precedes  a  series  of  learning  cues 
(which  are,  In  this  case,  Q9's). 

2)  Learning  cues.  These  cues  are  sequenced  so  that  the 
trainee  will  learn  the  desired  criterion  behaviors,  e.g.,  how  to 
write  a  SUM  function.  On  the  Training  Sequence  Logic  Flow  Dia¬ 
gram,  learning  cues  are  differentiated  according  to  their  train¬ 
ing  function  and  according  to  their  format. 

According  to  its  training  function,  which  may  change  from 
one  part  of  the  training  sequence  to  another,  a  learning  cue  Is 
identified  as  one  of  the  following  types:  LX,  PX,  RX,  LY,  PY, 

PEY,  Text,  CUM  TEXT,  RY,  CUM  REVIEW,  PRACTICE  Y,  and  CUM  PRACT 
PROB.  For  example,  on  page  1  of  Figure  8,  the  first  series  of 
learning  cues  is  represented  by  (  LX - 1 ) 

(lxED 

According  to  its  format,  a  learning  cue  is  identified  as 
one  of  the  following  types:  Q8,  Q9,  Qll,  Q19,  and  Q19A.  For 
example,  on  page  1  of  Figure  8,  the  first  series  of  learning 
cues  Is  Identified  as  a  series  of  Q9's  preceded  by  an  Instruc¬ 
tional  cue  (Q7). 

The  different  types  of  training  functions  for  learning  cues 
will  now  be  explained.  The  learning  cues  used  by  this  capability 
are  logically  and  sequentially  divided  into  two  major  groups: 

1)  The  non-QL  learning  cues.  These  cues  provide  a  basic 
foundation  for  the  course  but  do  not  teach  Query  Language  itself. 
They  are  divided  into  the  sets  called  MXM  SETS,  which  precede 
all  other  learning  cue  sets  used  by  this  capability.  Thus,  on 
page  1  of  Figure  o,  the  first  series  of  learning  cues  have  an  X 
subscript,  which  Identifies  them  as  non-QL  learning  cues. 

2)  The  QL  learning  cues.  These  cues  provide  trainees  with 
all  training  and  evaluation  materials  that  are  needed  In  order  to 
learn  to  use  Intermediate  Query  Language,  Model  II;  these  cues 
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are  divided  into  the  sets  called  "Y*1  SETS.  Thus,  on  page  2  of 
Figure  8,  the  first  series  of  learning  cues  have  a  Y  subscript, 
which  identifies  them  as  QL  learning  cues. 

In  each  non-QL  X  SET,  learning  cues  may  be  further  subgrouped 
according  to  their  sequence  and  function.  There  are  two  major 
sections  of  learning  cues  in  each  X  SET: 

1)  An  LX  section.  This  is  a  sequence  of  cues  (Q9's)  used 
primarily  fcr  training.  Each  IX  section  has  two  subsections: 
a)  the  initial  series  that  provides  basic  training,  and  b)  the 
final  series  that  reviews  the  information  taught  in  the  first 
section . 

2)  A  PX  section.  This  is  a  sequence  of  cues  (Q8's)  used 
primarily  for  evaluation  of  the  trainee's  proficiency  on  the 
information  covered  in  the  preceding  IX  section. 

In  each  X  SET,  these  two  sections  are  followed  by  computer 
error-analysis  of  the  trainee's  responses  on  the  PX  cues  and.  If 
necessary,  appropriate  remedial  work. 

Learning  cues  may  also  be  subgrouped  In  each  Y  SET  according 
to  their  sequence  and  function.  There  are  three  major  sections 
of  learning  cues  In  each  Y  SET: 

1)  An  LY  section.  This  is  a  sequence  of  cues  ( Q9 ' s )  used 
primarily  for  training.  Each  LY  section  ha3  three  subsections, 
in  this  general  order:  a)  the  basic  training  sequence,  called 
a  TEXT  sequence;  b)  the  sequence  that  reviews  the  Information 
taught  by  the  TEXT  sequence  --  this  Is  called  a  REVIEW  sequence; 
and  c)  a  series  of  cues  that  emphasize  the  development  of  QL 
statements  for  specified  data  retrieval  problems.  This  series 
is  intended  to  give  the  trainee  practice  In  using  the  QL  ele¬ 
ments  taught  in  the  TEXT  and  REVIEW  sequences.  This  Is  called 

a  PRACTICE  PROBLEM  sequence. 

2)  A  PY  section.  This  Is  a  series  of  cues  (Qll's)  used 
primarily  for  evaluation  of  the  trainee's  proficiency  on  the 
information  In  one  or  more  of  the  preceding  LY's.  The  com¬ 
plexity  of  the  computer  error-analysis  on  each  problem  is 
minimized  by  the  restriction  of  error-analysis  to  specified 
parts  of  the  appropriate  QL  statement. 

In  each  Y  SET,  these  two  sections  are  followed  by  computer 
error-analysis  of  the  trainee's  responses  on  the  PY  cues  and.  If 
necessary,  appropriate  remedial  work.  There  are  three  levels  of 
remedial  work,  appropriate  to  three  evaluation-ratings,  GOOD, 
AVERAGE  and  POOR.  In  general,  remedial  work  for  a  rating  of 
POOR  is  excerpted  from  a  TEXT  subsection  of  an  LY  section; 
remedial  work  for  a  rating  of  AVERAGE  is  excerpted  from  a 
REVIEW  subsection  of  an  LY  section;  and,  remedial  work  for  a 
rating  of  GOOD  Is  excerpted  from  a  PRACTICE  PROBLEM  subsection 
of  an  LY  section.  Since  remedial  work  in  an  area  Is  forced  on 
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a  trainee  in  Experimental  Group  A  if  his  PY  errors  on  that  area 
were  excessive,  any  remedial  sequence  taken  at  this  point  is 
called  a  FORCED  REMEDIAL  sequence. 

3)  Following  the  error-analysis  and  any  necessary  remedial 
work  for  a  Y  SET  is  a  PEY  section.  This  is  a  series  of  cues 
(Q19's  and  Q19A's)  that  give  the  trainee  practice  in  developing 
complete  QL  statements  for  specified  problems,  with  feedback  to 
the  trainee  indicating  the  correct  QL  statement  for  each  problem 
presented.  This  section  is  not  used  for  computer  evaluation;  it 
is  only  used  for  a  trainee's  self-evaluation.  Since  the  trainee's 
answers  in  this  section  are  not  evaluated  by  the  computer,  the 
answer  format  used  by  the  trainee  does  not  constitute  a  problem 
for  error-analysis.  Therefore,  the  trainee  types  the  entire  QL 
statement  as  his  answer,  not  Just  the  restricted  excerpts  required 
for  his  answers  in  the  PY  section.  Since  these  cues  provide  prac¬ 
tice  in  writing  QL  statements  and  the  answer  formats  are  not 
artificially  restricted  by  the  length  of  the  answer  required, 
these  cues  are  called  free-form  practice  exercises. 

For  each  Y  SET,  after  these  three  sections  are  complete, 
the  trainee  is  given  the  option  of  taking  remedial  work  in  one 
or  more  areas  for  which  the  trainee's  responses  in  the  last  PY 
section  were  evaluated.  Since  remedial  work  at  this  point  is 
taken  only  by  free-choice  and  consists  of  REVIEW  materials  from 
one  of  the  last  LY  sections,  a  remedial  sequence  that  the  trainee 
chooses  to  take  at  this  point  is  called  FREE-CHOICE  REVIEW.  As 
noted  earlier,  this  is  the  only  remedial  work  available  to  Experi¬ 
mental  Group  B. 

As  mentioned  previously,  the  X  and  Y  SETS  are  different  in 
terms  of  their  position  in  the  overall  training  sequence  and  their 
general  training  functions:  X  SETS  are  placed  first  and  emphasize 
non-QL  topics;  Y  SETS  are  last  and  emphasize  the  use  of  QL  ele¬ 
ments.  Cue  sets  may  also  be  differentiated  in  terms  of  their 
general  training  function  and  the  overall  scope  of  their  training- 
and-evaluat ion  materials.  In  terms  of  function  and  scope,  there 
are  four  types  of  sets: 

All  of  the  X  SETS  are  of  one  type: 

1)  Non-QL  PRETEST  sets.  These  precede  the  QL  materials 
that  are  tested  by  the  final  posttest;  in  each  set,  the  PX  sec¬ 
tion  tests  only  the  materials  covered  by  the  LX  section  in  the 
same  set. 

The  Y  SETS  include  three  different  types  of  sets: 

2)  QL,  INDEPENDENT  sets.  In  each  set,  the  LY  section  teaches 
new  materials,  and  the  PY  section  tests  only  the  materials  covered 
by  the  LY  section  in  the  same  set. 
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3)  QL,  CUMULATIVE  sets.  In  each  set,  the  LY  section  reviews 
the  materials  covered  in  the  INDEPENDENT  sets  that  followed  the 
last  CUMULATIVE  set,  and  the  PY  section  tests  all  materials  cov¬ 
ered  after  the  last  CUMULATIVE  Y  SET. 

4)  The  QL,  POSTTEST  set.  In  this  set,  the  LY  section  reviews 
the  materials  covered  throughout  all  of  the  sets.  The  PY  section 
is  the  posttest,  which  tests  all  materials  covered  over  all  of 
the  other  Y  SETS  in  the  program.  It  Is,  In  effect,  the  end-of- 
course  criterion  test. 

The  entire  program  consists  of  the  following  sets,  in  the 
order  described : 

1)  five  non-QL,  PRETEST  X  SETS 

2)  a  series  of  13  QL  INDEPENDENT  Y  SETS,  with  six  QL 
CUMULATIVE  Y  SETS  interpolated  at  appropriate  points 
in  the  overall  sequence 

3)  one  QL,  POSTTEST  Y  SET. 
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Fig.  8.  The  training  sequence  logic  flow  diagram 
for  computer  directed  training.  (sheet  1  of  4) 
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Fig.  3.  The  training  sequence  logic  flow  diagram 
for  computer  directed  training.  (sheet  2  of  4) 
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Fig.  S.  The  training  sequence  logic  flow  diagram 
for  computer  directed  training.  (sheet  3  of  4) 
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Fig.  S.  The  training  sequence  logic  flow  diagram 
for  computer  directed  training.  (sheet  4  of  4) 


The  proposed  program  will  provide  evaluative  data  that  are 
more  extensive  than  that  generally  obtained  by  non-computer-based 
training  studies.  Such  data  will  make  it  possible  to  relate  vari¬ 
ous  features  of  the  program  to  performance  measures.  The  evalua¬ 
tive  data  that  will  be  made  available  consists  of  the  following 
items : 


1.  Summary  data  for  all  trainees  who  have  completed  the 

program  at  any  time 

a.  the  date 

b.  the  total  number  of  trainees  who  have  completed 
the  program 

c.  the  total  number  of  trainees  still  in  progress 
on  the  program 

d.  the  average  time  to  complete  the  program:  for 
Experimental  Group  A  and  for  Experimental  Group  B 

e.  the  average  percent  error  on  the  program:  for 
Experimental  Group  A  and  for  Experimental  Group  B 

f.  the  average  percent  error  on  the  posttest:  (l)  on 
the  first  try  and  (2)  on  the  final  (second)  try:  for 
Experimental  Group  A  and  for  Experimental  Group  B 

g.  the  average  percent  error  on  each  subcategory  of  QL: 
(l)  averaged  over  all  evaluative  problems  except  the 
posttest  for  Experimental  Group  A  and  for  Experimental 
Group  E;  and,  (2)  averaged  over  the  posttest  for  all 
trainees . 


2.  Individual  trainee  data  for  any  trainee  except  a  profi¬ 
ciency  maintenance  trainee,  updated  for  any  point  during 

training  and  the  final,  updated  data  at  the  time  the 

trainee  completes  the  program 

a.  the  date  on  which  the  data  is  obtained 

b.  the  trainee's  name,  serial  number,  and  experimental 
group  (A  or  B) 

c.  the  cumulative  training  time  for  the  trainee  (at 
the  time  these  data  are  obtained) 

d.  the  trainee's  average  percent  error  over  all  evalua¬ 
tive  problem  sections  requiring  QL  compositions 
except  for  the  posttest 

e.  the  trainee's  average  percent  error  on  the  posttest: 

(1)  for  the  first  try  on  each  problem;  and,  (2)  for 
the  final  (second)  try  on  each  problem 

f.  the  trainee's  average  percent  error  on  each  of  the 
initial  problem  sections  that  do  not  require  com¬ 
posing  QL  statements 

g.  for  each  posttest  problem:  (1)  the  number  of  tries; 

(2)  the  possible  number  of  errors;  (3)  the  number  of 
errors  the  trainee  made  on  the  first  try;  (4)  the 
number  of  errors  the  trainee  made  on  the  second 
(final)  try 

h.  for  each  QL  subcategory:  (l)  the  average  percent 
error  for  the  trainee  over  all  QL-evaluat ive  problem 
sections  except  the  posttest  (l.e.,  during  training 
on  Query  Language);  (2)  the  average  percent  error 
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for  the  trainee  on  the  posttest;  (3)  the  number  of 
times  the  trainee  chose  that  subcategory  for  free- 
choice  review  over  the  entire  program 
1.  the  trainee's  sequence  of  remedial  materials  for 
the  last  20  training  sets.  Identified  as  either 
forced  remedial  or  free-choice  and  further  Identified 
by  the  point  at  which  these  materials  were  taken 
(i.e.,  Identified  by  the  evaluative  problem  section 
Immediately  preceding  each  remedial  set),  the  subcate¬ 
gory  for  which  they  were  given,  and  the  trainee's 
rating  on  that  subcategory.  Since  the  trainee's  use 
of  any  remedial  sequence  for  one  of  the  Initial,  non- 
QL  problem  sections  can  be  determined  by  whether  his 
percent  error  on  that  section  exceeded  5a  and  since 
there  is  only  one  remedial  sequence  for  each  of  the 
non-QL  problem  sections,  the  trainee's  path  through 
the  X  SETS  is  already  determined.  Since  the  basic 
(vs.  remedial)  training  sequence  of  the  Y  SETS  is 
known,  the  above  information  will  complete  the  set  of 
information  needed  to  determine  the  trainee's  entire 
path  through  the  program,  and  his  location  at  any  time 
prior  to  completion. 

3.  The  complete  QL  statements  typed  by  each  trainee  as 
answers  to  the  data-retrieval  problem  in  the  practice 
exercises . 

4.  The  complete  QL  statements  typed  by  each  trainee  as 
answers  to  the  posttest  problems  and  the  parts  of  these 
statements,  as  typed  by  the  trainee,  that  underwent  error 
analysis . 

The  comparison  between  Experimental  Groups  A  and  B  plus  an 
analysis  of  the  various  types  of  data  made  available,  as  described 
above,  would  not  only  provide  information  needed  to  make  revisions 
in  the  training  logic  that  would  improve  the  overall  efficiency 
and  effectiveness  of  the  computer  directed  training  (CDT)  program, 
but  such  information  would  also  increase  the  general  level  of 
knowledge  with  respect  to  the  potential  role  of  computer-based 
instructional  systems  in  the  military  environment.  Thus,  if  the 
CDT  program  is  shown  to  be  an  effective  teaching  instrument,  this 
computer  directed  training  capability  could  be  used  advantageously 
for  teaching  the  use  of  other  System  4?3L  capabilities  and  as  a 
general  model  for  apolication  to  training  for  other  computer- 
based  systems. 

3.3.2  Operating  Procedures  and  the  Procedural  Flow  Diagram.  Any 
operator-processor  communication  for  System  473L  requires  knowledge 
of  the  integrated  console  operating  procedures.  These  are 
described  in  an  Air  Force  document,  473L-OS-43:  Operational 
Specification  for  the  Integrated  Console,  Model  II.  In  addition, 
each  capability  will  employ  a  separate  overlay  that,  when  used  with 
the  integrated  console  process  step  keys,  will  present  to  the  opera¬ 
tor  the  capability  options.  The  function  of  each  process  step 
key  associated  with  the  overlay  for  the  computer  directed  train¬ 
ing  capability  is  shown  in  Table  6  of  this  report. 


TABLE  6 


Design  for  Computer-Directed  Training  Overlay- 
Process  Step  Key  Functions  (Page  1  of  3) 


Key  Designation 


LI  _ 

TRAINEE 


L2  _ 

INSTRUCTOR 


PROFICIENCY 

MAINTENANCE 


L7  _ 

SUMMARY 


L8 


INDIVIDUAL 

TRAINEE 


UPDATE 

TRAINING 

MATERIAL 


LIO 


FREE 

CHOICE 


Function 

This  key  signals  the  program  that  the  operator 
wants  to  start  or  continue  training  on  the  CDT 
program.  When  this  key  is  pressed,  the  Name 
and  Serial  Number  Entry  cue  (Q6)  appears  on 
the  display  screen. 

This  key  signals  the  program  that  the  operator 
wants  to  either  obtain  trainee  data  (for  print¬ 
out  or  deletion)  or  to  update  training  materials. 
When  this  key  is  pressed,  the  Directions  for 
Obtaining  Data  or  Updating  cue  (Q3)  appears  on 
the  display  screen. 

This  key  signals  the  program  that  the  operator 
wants  some  of  the  CDT  materials  tnat  are  avail¬ 
able  for  proficiency  maintenance.  When  this 
key  is  pressed,  the  Name  and  Serial  Number  Entry 
cue  (Q6)  appears  on  the  display  screen. 

This  key  signals  the  program  that  the  operator 
wants  P/O  1,  the  line  printer  printout  of 
Summary  Data  for  all  trainees  (except  profi¬ 
ciency  maintenance  trainees).  When  this  key 
is  pressed,  printout  #1  will  be  printed  out. 

This  key  signals  the  program  that  the  operator 
wants  to  obtain  data  for  an  individual  trainee, 
either  on  a  printout  or  for  deletion.  When  this 
key  is  pressed,  the  Directions  for  Obtaining  or 
Deleting  Data  on  an  Individual  Trainee  cue  (Q5) 
appears  on  the  display  screen. 

This  key  signals  the  program  that  the  operator 
wants  to  update  some  of  the  training  materials 
(i.e.,  the  content  of  some  of  the  cues). 


This  key  signals  the  program  that  the  operator 
wants  to  take  remedial  work  in  one  or  more  areas. 
When  this  key  is  pressed,  either  the  Index  for 
Free-Choice  Review  for  Trainees  Who  Have  Not 
Completed  the  Program  (Q2l)  or  the  Index  for 
Free-Choice  Review  for  Trainees  Who  Have  Com¬ 
pleted  the  Program  (Q4)  will  appear  on  the  dis¬ 
play  screen,  depending  on  whether  the  trainee 
has  completed  the  posttest  or  not. 
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TABLE  6  (cont.) 
(Page  2  of  3) 

Key  Designation 

Lll 


POSTTEST 


L12 


TRY 

AGAIN 


L13 


ERROR 

MATRIX 


LI  4 _ 

EVALUATION 

CONTINUATION 

MATRIX 


L15  _ _ 

CUE 

FILE 

MATRIX 


Ll6  _ 

CRITERIA 

MATRIX 


L17 


FLEX 

COURSE 


Function 

This  key  signals  the  program  that  a  proficiency 
maintenance  trainee  wants  to  take  the  posttest 
(the  very  last  PY  section,  In  the  last  set,  Y  = 
20).  When  this  key  Is  pressed,  the  first  cue 
In  the  posttest  (a  Qll)  will  appear  on  the  dis¬ 
play  screen. 

This  key  signals  the  program  that  the  operator 
wants  to  take  the  last-presented  problem  again. 
When  this  key  is  pressed,  the  last  problem  in 
the  FY  section  being  worked  on,  which  is  a 
problem  cue  requiring  computer  error-analysis  of 
the  trainee’s  answer(s)  (a  Qll), will  re-appear 
on  display  screen. 

This  key  signals  the  program  that  for  updating 
purposes,  the  instructor  wants  to  obtain  P/oS, 
the  Possible  Number  of  Errors  on  Each  Subcate¬ 
gory  for  Each  PY  Section;  on  Each  PX  Section; 
and  on  Each  Posttest  Problem.  When  this  key  is 
pressed,  printout  #8  will  be  printed  out  on  the 
line  printer. 

This  key  signals  the  program  that  for  updating 
purposes,  the  instructor  wants  to  obtain  P/07, 
Remedial  Continuations.  When  this  key  Is  pressed, 
printout  #7  will  be  printed  out  on  the  line 
printer . 

This  key  signals  the  program  that  for  updating 
purposes,  the  Instructor  wants  to  obtain  P/05, 
Cues  (the  contents  for  all  cues).  When  this 
key  is  pressed,  printout  #5  will  be  printed  out 
on  the  line  printer. 

This  key  signals  the  program  that  for  updating 
purposes,  the  instructor  wants  to  obtain  P/o6, 
Criteria.  When  this  key  is  pressed,  printout 
^6  will  be  printed  out  on  the  line  printer. 

This  key  signals  the  program  that  for  updating 
purposes,  the  Instructor  wants  to  obtain  P/0£ , 
FLEX  COURSE,  which  specifies  the  number  of 
PRETEST  (X)  SETS,  the  number  of  POSTTEST  prob¬ 
lems,  and  the  number  of  subcategories.  When 
this  key  Is  pressed,  printout  #9  will  be  printed 
out  on  the  line  printer. 
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TABLE  6  (cont.) 
(Page  3  of  3) 


Key  Designation  Function 

L25  This  key  signals  the  program  that  either  the 

operator  does  not  wish  to  exercise  any  special 
training  options  available  at  this  time  or, 
none  are  available.  When  this  key  is  pressed, 
the  next  cue  that  is  available  in  queue  (with¬ 
out  branching  out  to  pick  up  cues  that  are  not 
presently  stacked  up)  will  appear  on  the  dis¬ 
play  screen.  The  type  of  cue  that  is  presented 
will  vary  from  one  training  point  to  another. 


CONTINUE 


L27  This  key  signals  the  program  that  the  operator 

does  not  wish  to  complete  the  remedial  sequence 
on  which  he  is  now  working.  When  this  key  is 
pressed,  the  trainee  will  be  sent  to  the  same 
continuation  point  that  he  would  have  reached 
had  he  completed  the  remedial  sequence:  if  the 
trainee  is  still  in  training  on  the  CDT  program 
and  he  has  not  yet  taken  the  posttest,  or  if  he 
is  a  proficiency  maintenance  trainee,  Q20  will 
appear  on  the  display  screen;  if  the  trainee  has 
just  completed  the  CDT  program  including  the 
posttest,  Q23  will  appear  on  the  display  screen. 


SKIP 


L29  This  key  signals  the  program  that  the  trainee 

is  going  to  stop  work  on  the  CDT  program  at  the 
present  time.  When  this  key  is  pressed,  Q32 
will  appear  on  the  display  screen,  and  the 
storage  of  data  for  a  trainee  on  the  work  done 
so  far  on  a  set  will  continue  until  storage  is 
complete . 


TERMINATE 

TRAINING 


A  diagram  of  the  training  sequence  in  terms  of  the  presenta¬ 
tion  of  specific  cues  and  capability  options  is  given  in  the 
Procedural  Flow  Diagram  for  the  Computer  Directed  Training  Capa¬ 
bility.  This  may  be  referenced  in  the  Operational  Specification 
for  Computer  Directed  Training  in  Intermediate  Query  Language, 

Model  II,  for  System  473L10! 


reference 


4. 
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3.4  DISCUSSION  AND  RECOMMENDATIONS 


The  techniques  for  a  computer  Directed  training  program 
briefly  described  here,  and  more  fully  described  in  the 
Operational  Specification,  are  sufficiently  general  as  to  be 
applicable  to  other  programs  of  training  in  artificial  languages 
that  are  relatively  fixed  in  format  and  vocabulary  (such  as 
computer  languages).  However,  it  is  recognized  that  other  com¬ 
puter  systems  may  not  have  available  the  type  of  equipment  for 
which  this  program  was  designed.  The  only  major  feature  of  the 
instructional  system  that  may  not  be  commonly  available  on  other 
systems  is  the  overlay  keyboard .  The  overlay  system  on  the 
integrated  console  was  not  originally  designed  to  perform  an 
instructional  function,  but  it  provides  an  excellent  opportunity 
for  permitting  limited  control  of  the  instructional  process  by 
the  trainee  and  increases  the  scope  of  interaction  at  the  com¬ 
puter  interface.  However,  this  overlay  keyboard  is  not  essen¬ 
tial  for  a  computer  directed  training  capability.  The  functions 
performed  with  the  overlay  could  be  simulated  by  outputs  on  a 
display  screen  or  typewriter  providing  a  trainee  with  choices 
that  would  have  been  indicated  on  the  overlay.  Although  this 
method  may  be  less  economical,  it  would  still  be  functional. 

It  is  felt  that  the  major  advantages  of  the  CDT  instruc¬ 
tional  system  are  the  periodic  analysis  of  the  trainee's  input 
and  the  selection  of  appropriate  remedial  materials  based  on 
this  analysis,  the  extensive  capabilities  for  output  data,  and 
the  introduction  of  some  learner  control  over  the  selection  of 
training  materials.  The  latter  feature  would  be  especially 
valuable  for  a  proficiency  maintenance  training  sequence,  where 
individual  needs  would  vary  greatly. 

Proficiency  maintenance  is  a  particularly  critical  training 
problem  with  command  and  control  systems  where  there  is  a  need 
for  highly  proficient  personnel  at  all  times  but  where  there  is 
little  opportunity  for  actual  practice  in  real  or  simulated 
situations.  It  would  be  possible  to  expand  the  computer  directed 
training  course  described  here  to  provide  frequent  realistic 
combat  exercjses  and  to  provide,  on  the  basis  of  an  analysis  of 
errors  made,  a  series  of  training  exercises  uniquely  designed 
to  build  proficiency  in  weak  areas.  This  approach  could  also 
be  generalized  to  other  command  and  control  systems  where  pro¬ 
ficiency  must  always  be  at  a  maximum  but  where  actual  practice 
may  be  at  a  minimum. 

Looking  further  into  tie  future,  it  would  be  possible  to 
build  into  a  CDT  program  appropriate  statistical  tests  that 
would  determine  the  effectiveness  of  the  program  as  measured 
against  appropriate  criteria  of  terminal  performance.  With 
initial  entrance  data  (e.g.,  initial  aptitude,  1^,  etc.)  and 
performance  data  collected  at  appropriate  points  throughout  the 
pregram,  statistical  computations  could  be  performed  after  an 
appropriate  N  of  trainees  had  been  obtained.  The  results  of 
such  an  analysis  may  suggest  changes  in  the  program,  e.g.,  a 


new  organization  cf  the  materials  and/or  a  shift  in  the 
parameters  used  for  deciding  cn  the  frequency  and  nature  of 
the  correctional  training  sequences.  Adjustments  of  this 
sort  could  be  accomplished  automatically  and  internally  by 
the  computer  itself.  Such  a  training  system  would  be  truly 
adaptive  in  that  it  would  adjust  its  own  training  strategy 
based  on  the  measured  success  of  the  program  being  usei  . 

For  example,  in  the  program  described  in  this  report,  the 
two  experimental  groups  (A  and  B)  could  be  compared  in  an 
analysis  of  variance  design  and  if  one  group  was  always 
inferior,  that  group  could  be  discarded,  or,  if  a  significant 
interaction  was  found  between  the  experimental  group  and  seme 
other  variable,  e.g.,  some  background  factor,  a  decision  logic 
cculd  be  internally  prepared  for  deciding  who  got  the  "A"  con¬ 
dition  and  who  got  the  "fl"  condition.  For  example,  perhaps  it 
would  be  shown  by  the  analysis  that  trainees  with  previous 
experience  with  computer  languages  should  be  given  the  program 
allowing  self  choice  (i.e.,  optional  remedial  sequences), 
whereas  those  trainees  with  no  previous  ex\  erience  in  the 
field  should  be  given  the  "lock-step”  program  (i.e.,  the  pro¬ 
gram  with  forced  remedial  sequences).  Entering  trainees,  in 
this  case,  would  be  asked  to  Indicate  this  information  and 
would  be  assigned  by  the  computer  to  the  appropriate  program . 

Other  relevant  "questions"  of  this  kind  could  be  "asked" 
in  an  effort  to  devise  CDT  systems  tnat  achieve  a  truly 
optimal  relationship  between  level  of  proficiency  and  time  in 
training,  These  approaches  are  within  the  present  "state- 
of-art."  They  are  expensive  and  they  are  yet  to  be  proven  by 
the  test  of  practical  experience.  In  spite  of  the  fact  that 
computer-directed  training  is  still  in  an  experimental,  neophyte 
phase,  the  military  may  provide  the  only  feasible  environment 
in  which  the  effectiveness  of  3uch  approaches  can  be  demonstratet 
The  existence  and  availability  of  large  capacity  computers,  the 
number  of  men  to  be  trained,  the  complexity  of  the  subject 
matter,  the  difficulty  of  obtaining  qualified  Instructors,  and 
the  critical  nature  of  the  training  requirements,  all  exist  in 
the  context  of  military  training  as  perhaps  in  no  other  sector 
of  the  training  world.  As  has  happened  in  so  many  other  areas, 
a  significant  general  advance  in  the  technology  of  training 
could  be  initiated  in  support  of  military  requirements.  The 
impact  of  such  an  advance  on  both  military  and  civilian  training 
would  be  hard  to  overestimate.  The  implementation  of  the  CDT 
program  described  in  this  report  would  be  an  important  step  in 
this  direction.  Further  refinements  of  the  approach  described 
here,  applications  to  other  military  training  problems,  and 
inclusion  of  more  sophisticated  strategies  such  as  have  been 
suggested  in  this  section  of  the  report,  would  represent  a  pro¬ 
gram  of  both  short-  and  long-term  value  to  military  and  civilian 
training11. 


■'■■'■See  reference  #6. 
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APPENDIX  A 


Number  of  Frames  in  Each  Volume  of  the  Program12 


Volume 

Number 

** 

First  I 
Tryout  II 

III 

IV 

V 


< 


VI 


VII 


VIII 

IX 


Number  of 


Section  of  Program_ Frames 


Preprogram  3^ 

General  Orientation  133 

Organization  of  Information  in  the  43 

Data  Files 

Selection  of  the  Proper  File  Indicator  21 

Selection  of  the  Correct  Attributes  and  51 

Values  Using  the  Data  File  Contents 
Manual 

Writing  Simple  Query  Language  Statements  -  88 

The  Simple  Qualifier 

Writing  Simple  Query  Language  Statements  -  55 

Writing  the  Output  Director  and  Output 
Selector 

Writing  Query  Language  Functions  -  Special  67 

Features  of  the  Qualifier 
Writing  Query  Language  Functions  -  Special  64 

Features  of  the  Selector 


Second 

Tryout 


XV  Titles  18 

XVI  Sorting  58 

XVII  Compound  Qualifiers  46 

XVIII  SAVE  Procedures  68 

XIX  CHECK  8 

XX  Review  and  Practice  of  II-XIX  73 

XXI  Use  of  the  Data  Control  Manuals  and  58 

Complex  Queries 

XXII  Complex  Queries  -  Computed  Attributes  74 

and  Special  Uses 

XXIII  Final  Practice  17 


X 

GCD  Functions 

64 

XI 

MIN  and  MAX  Functions 

137 

XII 

GREATEST  and  LEAST 

51 

XIII 

Simple  SUMS 

118 

XIV 

More  on  SUM 

62 

TOTAL  NUMBER 


1448 


IP 

There  were  no  frames  in  Volumes  XXIV  through  XXIX;  these 
volumes  contained  reference  or  test  materials  only. 


APPENDIX  B 


Final  Test* 


ALL  MANUALS,  EXHIBITS,  AND  REVIEW  PANELS  MAY  BE  USED. 


A.  General  Information 

1.  For  what  purpose  is  the  473L  system  designed? 


2.  Determine  what  file  indicator  you  would  use  to  obtain 
the  following  information  (you  may  use  Exhibits  2N, 

3A  and  3B ) s 

a.  Specific  data  concerning  current  materiel  inventory 

b.  The  types  of  aircraft  possessed  by  TAC 

c.  Detailed  characteristics  of  tactical  aircraft 

d.  The  selection  of  a  specific  plan  that  fits  certain 
requirements 


Label 

the  following  outputs 

with  the 

appropriate 

output 

director. 

UNIT 

MDS 

AFLD  NAME 

ANDREWS 

AMARILLO 

TRUAX 

23FIS 

f86a 

STATE 

MD 

TEX 

WIS 

F100F 

RNWY  LENGTH 

12000 

13500 

9000 

F104B 

11000 

10000 

8500 

113FIS 

F100B 

$000 

5500 

7000 

F101C 

8000 

F105D 

4. 


In  the  Query  below,  use  brackets  and  labels  to  Indicate 

a.  The  program  Indicator 

b.  The  file  indicator 

c .  The  qualifier 

d.  The  output  director 

e.  The  output  selector 

f.  The  qualifier  conjunction 

g.  The  output  director  conjunction 

h.  Modifiers 

i.  Attributes 

j .  Comparators 

k.  Values 


RETRIEVE 

PLAN  IDENT 

WITH 

PLAN  CAT 

=  EX, 

VUN  PLAN 

IND  =  NO 

THEN 

PRINT 

PLAN  IDENT, 

TYPE  OP, 

START 

DATE  "| 

*To  save  space,  answer  blanks  are  omitted  in  this  reproduction 
of  the  test. 
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5.  Identify  with  arrows  the  points  at  which  CHECK  could  be 

Inserted  in  the  following  QL  statement. 

RETRIEVE  AIRFIELDS  WITH  COUNTRY  =  CANADA 

THEN  PRINT  AND  COUNT  AFLD  NAME  *  AFLDS  IN 

CANADA  *“! 

6.  Match  the  following: 

- AND  a.  commas  between  modifiers 

_ OR  b.  semicolons  between  modifier  sets 

7.  Select  the  correct  statements  about  SUM. 

a.  Only  one  SUM  expression  may  be  used  in  a  QL  qualifier. 

b.  Only  one  SUM  expression  may  be  used  in  a  QL  selector. 

c.  If  more  than  one  SUM  expression  is  used  in  a  QL 
statement,  all  SUM  expressions  must  have  (or  repre¬ 
sent)  the  same  set  of  SUM  control  attributes. 

d.  If  more  than  one  SUM  expression  is  used  in  a  QL 
statement,  all  SUM  expressions  must  have  (or  repre¬ 
sent)  the  same  set  of  summed  attributes. 

e.  A  SUM  function  used  in  one  subordinate  query  of  a 
complex  Query  Language  statement  automatically  applies 
to  all  subordinate  Queries  in  the  statement. 

f.  A  SUM  function  used  in  one  modifier  set  of  a  compound 
qualifier  automatically  applies  to  all  modifier  sets 
of  the  compound  qualifier. 

g.  The  only  SUM  expression  that  may  be  retained  is  SUM, 
referencing  a  SUM  already  computed  in  the  qualifier. 


B.  Problems 

Write  the  appropriate  Query  Language  statements  for  the  problem 
below  (you  may  refer  to  any  manuals  or  Exhibits). 

1.  Write  a  Query  to  print  out  the  entire  UNIT/LOC  STRNGTH  file 
sorted  in  descending  alphabetic  order  according  to  the  name 
of  the  installation.  Specify  also  that  the  following  SUMs 
are  to  be  printed  out: 

a.  for  each  LSA,  the  number  of  U.  S.  citizens. 

b.  for  the  entire  file,  the  total  number  of  U.  S.  citizens 

2.  You  are  checking  the  data  files  to  determine  whether  some 
information  has  been  omitted.  Determine  which  planned 
operating  bases  have  no  values  for  ITEM  in  the  files. 
Request  a  horizontal  printout. 

3.  Obtain  the  plan  idents  of  all  exercise  plans  that  require 
C119A  aircraft  and  are  scheduled  to  start  on  or  between 

1  June  1966  and  15  July  1966.  Use  a  horizontal  display. 


APPENDIX  B  (cont.) 

4.  Store  a  Query  which  will  print  out  the  number  of  authorized 
weapons  systems  for  a  force  (UNIT,  UNIT  LOC,  MDS  and  COMD) 
to  be  specified  at  a  later  time.  Leave  room  for  other 
attributes  in  the  selector. 

5.  Write  a  Query  to  display  the  numbers  of  any  SAVEd  state¬ 
ments  pertaining  to  F105G  entries  in  FORCE  STATUS. 

6.  Make  the  necessary  entries  to  complete  this  Query  for 
Barclay  Air  Force  Base.  Add  the  title  "Barclay  Status 
Data"  to  the  output. 

RETRIEVE  MATERIEL  STATUS  WITH  POB  =  '1'  THEN  PRINT  ALL1 

/Incomplete  query  -  to  complete  type  insert  number  and  = 

SYMBOL  FOLLOWED  BY: 

ADDITIONAL  ATTRIBUTES 

ADDITIONAL  VALUES 

OTHER  PORTIONS  OF  QUERY  STATEMENT  AS  DESIRED. 

START  FOR  EACH  NUMBER  IN  STATEMENT  OR  ERASE  COMPLETE 
INSERT  NUMBER  (E.G.,  '3').  TYPE  EOM  AFTER  LAST  INSERT. 

VCHECK  PUNCTUATION  AND  FORMAT  BEFORE  PRESSING  ENTER. 

(1  =  ' 

7.  Write  a  Query  to  select  either  airfields  in  Canada  that 
are  no  more  than  500  nautical  miles  away  from  Hill  and 
can  land  or  depart  (whichever  is  lower)  at  least  10  air¬ 
craft  per  hour  under  IFR  conditions  or  airfields  in  the 
United  States  that  are  no  more  than  350  nautical  miles 
from  KANSAS  CITY  and  have  the  above  arrival/departure 
restrictions.  Specify  a  horizontal  printout  of  airfield 
names,  the  country,  the  GCD  from  both  airfields,  and  the 
number  of  aircraft  that  can  be  landed  or  departed 
(labeled  MIN  ARR/DEP). 

8.  Write  a  Query  to  print  out  the  names  and  GCD's  from 
Bailey  of  all  airfields  that  are  no  further  from  Bailey 
than  Peterson  is  from  Wallace.  Eliminate  Bailey, 

Peterson,  and  Wallace  from  being  selected. 

9*  Write  a  Query  to  find  the  LOG  ABSTRACTS  annex  to  plan 

32111  and  print  out  the  annex  plan  ident  and  the  abstract. 
Also,  a  count  of  the  number  of  and  the  plan  idents  of 
other  plans  using  this  annex  is  desired.  Specify  a 
printout . 

10.  Write  a  Query  to  select  the  medical  plan(s)  with  the 

fewest  total  number  of  patients  per  day  to  be  moved  from 
the  collect  bases.  Specify  a  vertical  display  of  the 
plan  ident,  the  total  number  of  patients  per  day  to  be 
moved,  and  all  strategic  data  related  to  the  route  seg¬ 
ment  requiring  the  fewest  number  of  AME  crews  per  day 
at  the  staging  base. 
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11.  Write  a  Query  to  find  the  closest  airfield  to  SEMHOI 
(excluding  SEMHOI)  that  Is  a  POB  for  at  least  10,000 
pounds  of  gaseous  nitrogen.  Obtain  a  vertical  display 
of  the  airfield  name,  the  GCD  from  SEMHOI,  and  the 
amount  of  gaseous  nitrogen  Intended  for  use  there. 

12.  Write  a  Query  to  find  those  FORCE  STATUS  entries  with 
minimum  ready  weapons  systems  of  at  least  10.  Entitle 
the  minimum  RDY  WP  SYS.  Specify  a  vertical  display  of 
the  force  and  the  weapons  system. 

13.  Write  a  Query  to  find  the  total  amount  of  jet  fuel  stored 
at  Parkersburg  (regardless  of  the  POB  of  the  fuel)  and 
those  airfields  In  logistics  subarea  3A  capable  of  storing 
at  least  that  much  jet  fuel.  Specify  a  horizontal  listing 
of  the  amount  of  jet  fuel  stored  at  Parkersburg,  the  names 
of  the  qualifying  airfields,  their  GCD's  from  Parkersburg, 
and  their  jet  fuel  capacities. 

14.  Write  a  Query  to  display  the  number  of  airfields  In  the 
United  States  having  all  navigational  aids  either  avail¬ 
able  or  estimated  and  having  at  least  1  of  the  lighting 
facilities  either  available  or  estimated. 

15.  Write  a  Query  to  find  all  plan  ldents,  the  POB's,  the 
MDS's,  and  the  number  of  airmen  deployed  from  a  POB  and 
MDS  set  (such  as  F105G's  at  Bailey)  when  at  least  1000 
airmen  are  to  be  deployed  from  a  set  and  the  POB  is  in 
the  United  States  or  when  at  least  1000  men  are  to  be 
deployed  from  a  set  and  the  airmen  are  from  an  RFG  unit. 
Specify  a  sort  in  ascending  order  according  to  MDS. 

16.  Write  a  Query  to  find  the  minimum  authorized  weapons 
system  of  B52A's  for  the  8lBW  unit  of  TAC  at  Parkersburg 
(entitled  WPSYS  AUTH)  and  those  plans  which  require  no 
more  crewed  B52A's  than  the  authorized  weapons  system. 
Specify  a  horizontal  display  of  the  weapons  system,  and 
the  plan  ldents  of  qualifying  plans. 


APPENDIX  C 


Final  Test  Scoring  and  Answer  Key 


Scoring  of  QL  Statements  on  the  posttest  is  based  on: 


1.  Program  director  —  1  point 

2.  File  indicator  —  1  point 

3.  WITH  —  1  point 

4.  Each  modifier  —  1  point 

5.  THEN  —  1  point 

6.  Output  director  —  1  point 

7.  Each  phrase  in  selector*  —  1  point 

8.  Punctuation  and  spelling  of  entire  QL  statement  —  1  point 


*  A  phrase  may  consist  of  a  single  attribute,  a  computed  attribute,  an 
attribute  =  INCR/DECR,  or  a  SAVE  instruction.  For^-example, 

e  q  &  Q)  , — 

...THEN  LIST  AND  COUNI^AFLD  NAME,  GCD,  GCD  (BEALE),  (£) 


IQUNT^AF 


''SUM  AND  TOTAL  BY  AFLD  NAME  (RNWY  LENGTH  =  INCR7?  MIN  (MAX  IFR  ARR, 


<2> 


MAX  IFR  DEP),  TITLE  =  MIN  TRAFF,  ^CALIFORNIA  AIRFIELD  INFO* 

^0 _ . 

SAVE  BY  SMITH  19 66  — | 


would  be  worth  12  points. 


Since  most  Query  Language  problems  can  be  solved  in  more  than  one  way, 
it  is  important  that  the  test  be  scored  by  someone  knowledgeable  in  Query 
Language.  Each  Query  should  first  be  assigned  a  number  of  possible  points  , 
according  to  how  the  student  attempted  to  solve  the  problem,  and  then  points 
subtracted  for  the  items  missed.  For  example,  if  a  student  wrote  the  Query 

RETRIEVE  AIRFIELDS  WITH  AFLD  NAME  =  PETERSON  THEN  RETAIN  GCD  (WALLACE): 

RETRIEVE  AIRFIELDS  WITH  AFLD  NAME  4  BAILEY  AND  PETERSON  AND  WALLACE, 

GCD  ( BAILEY <[R1,  GCD  (WALLACE),  OR]  THEN  PRINT  AFLD  NAME,  GCD  ( BAILEY H 

for  question  number  8,  he  would  receive  17  possible  points,  and  lose  two: 
l)  for  using  AND1 s  in  the  AFLD  NAME  4  modifier,  and  2)  he  would  lose  his 
punctuation  and  spelling  point  for  using  parentheses  around  WALLACE  in  the 
value  [Rl,  GCD  WALLACE,  OR]. 

It  will  be  necessary,  for  each  student,  to  total  the  possible  points  and 
the  correct  points  and  compute  his  percentage. 
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TEST  ANSWER  KEY 


A.  General  Information 

1.  provide  information  needed  for  decision-making 

2.  a.  MATERIEL  STATUS 


Points 

1 

1 


b .  FORCE  STATUS 


c .  TACTICAL  CHAR 


d.  PLAN  IDENT 


3.  LISTH  OR  PRINTH  OR,  HORIZONTAL 
LISTV  OR  PRINTV  OR,  VERTICAL 

h.  In  the  Query  below,  use  brackets  and  labels  to  indicate: 

a.  The  program  indicator 

b.  The  file  indicator 

c.  The  qualifier 

d.  The  output  director 

e.  The  output  selector 

f.  The  qualifier  conjunction 

g.  The  output  director  conjunction 

h.  Modifiers 

i .  Attributes 

j .  Comparators 

k.  Values 


(a) 

program 

indicator 


RETRIEVE 


mo> 


PLAN  IDENT 

Jh) 

defier _ _ 


(f) 

qualifier 

conjunction 

WITH 


11 


(c) 

qualifier 


(h) 

modifier 

— 


^TTT  u)  (ITJ 

attribute  comparator  value 


PLAN 


CAT 


EX, 


attribute  ( i )  comparator ( j )value(k) 
VUN  PLAN  IND  =  NO 


output  director 
conjunction(g) 

THEN 


output 

director(d) 


PRINT 


S'"  attribute  (i) 
PLAN  IDENT, 


output  select or(e) 


attribute  (i) 


TYPE  OP, 


attribute  (Tj^ 
START  DATE 


5.  RETRIEVE  AIRFIELDS  WITH  COUNTRY 

*  t 


CANDA  THEN 

♦ 


PRINT  AND  COUNT  AFLD  NAME  *AFLDS  IN  CANADA*  .  *1  6 

♦  t  ♦ 
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Points 

6.  a  1 

b  1 

7-0  1 

b.  1 

1 

d.  1 

e.  1 

CD 

g-  i 

B.  Problems 

1.  RETRIEVE  UNIT/LOC  STRNGTH  THEN  PRINT  INSTALLATION  =  DECR ,  ALL, 

SUM  AND  TOTAL  BY  LSA  (TOTAL  US  CIVILS)  ~i  8 

2.  RETRIEVE  MATERIEL  STATUS  WITH  ITEM  =  BLANK  THEN  PRINTH  POB *n  8 

3.  RETRIEVE  PLAN  IDENT  WITH  PLAN  CAT  =  EX,  MDS  =  C119A,  START  10 

DATE  >  01JUN66  AND  <  1XTUL66  THEN  LISTH  PLAN  IDENT  “1 

U.  RETRIEVE  FORCE  STATUS  WITH  UNIT  =  ?,  UNIT  LOC  =  ? ,  MDS  =  ?,  13 

COMD  =  ?  THEN  PRINT  MIN  (CRWS  AUTH ,  AC FT  AUTH),  ?  SAVE  ~i 

5.  FIND  FORCE  STATUS,  F105G  ->  k 

6.  RETRIEVE  MATERIEL  STATUS  WITH  POB  =  '1'  THEN  PRINT  '2'  U  2 

(1  =  'BARCLAY  2  =  'ALL  * BAR CLAY  STATUS  DATA* '  -»  2 

7.  RETRIEVE  AIRFIELDS  WITH  COUNTRY  =  CANADA,  GCD  (HILL  <  500),  18 

MIN  (MAX  IFR  ARR,  MAX  IFR  DEP)  >  10,  TITLE  =  MIN  ARR/DEP ; 

COUNTRY  =  USA,  MIN  (MAX  IFR  ARR,  MAX  IFR  DEP)  >  10,  TITLE  = 

MIN  ARR/DEP,  GCD  (KANSAS  CITY  <  350)  THEN  PRINTH  AFLD  NAME, 

COUNTRY,  GCD,  MIN  ARR/DEP  (OR  PRINTH  AFLD  NAME,  SAME  “0 

8.  RETRIEVE  AIRFIELDS  WITH  AFLD  NAME  4  BAILEY  OR  PETERSON  OR  9 

WALLACE,  GCD  (BAILEY  <  GCD  (PETERSON,  WALLACE))  THEN  PRINT 

AFLD  NAME,  GCD  ^ 
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Points 

9.  RETRIEVE  PLAN  IDENT  WITH  PLAN  IDENT  =  32111  THEN  RETAIN  2k 

LOG  ABST  IDENT:  RETRIEVE  LOG  ABSTRACTS  WITH  PLAN  IDENT  = 

[Rl,  LOG  ABST  IDENT,  OR]  THEN  PRINT  PLAN  IDENT,  LOG  ABSTR : 

RETRIEVE  PLAN  IDENT  WITH  LOG  ABST  IDENT  =  [Rl,  LOG  ABST 
IDENT,  OR]  THEN  PRINT  AND  COUNT  PLAN  IDENT  -\ 

10.  RETRIEVE  MED  PLAN  RQMT  WITH  SUM  BY  PLAN  IDENT  (COLLECT  CSU  10 
RQMT  =  MIN),  STAGE  AME  RQMT  =  LEAST  THEN  LISTV  SUM,  RTE  ID  = 

GROUP  “I 

11.  RETRIEVE  MATERIEL  STATUS  WITH  POB  f  SEMHOI,  ITEM  =  NITRO  GAS,  28 
TOTAL  ASSETS  >  10000  THEN  RETAIN  POB:  RETRIEVE  AIRFIELDS  WITH 
AFLD  NAME  =  [Rl,  POB,  OR],  GCD  ( SEMHOI  =  MIN)  THEN  RETAIN  AND 
LISTV  AFLD  NAME,  GCD:  RETRIEVE  MATERIEL  STATUS  WITH  ITEM  = 

NITRO  GAS,  POB  =  [R2,  AFLD  NAME,  OR]  THEN  LISTV  TOTAL  ASSETS 

12.  RETRIEVE  FORCE  STATUS  WITH  MIN  (CRWS  RDY ,  ACFT  RDY)  >  10,  TITLE  13 

=  RDY  WP  SYS  THEN  LISTV  COMD,  UNIT,  UNIT  LOC,  MDS,  RDY  WP  SYS "1 

13.  RETRIEVE  MATERIEL  STATUS  WITH  ITEM  =  JET  FUEL,  STORAGE  LOC  =  20 

PARKERSBURG  THEN  RETAIN  AND  LISTH  SUM  (SL  ASSET):  RETRIEVE 
AIRFIELDS  WITH  LSA  =  3A,  JET  FUEL  CAP>  [Rl,  SUM  SL  ASSET,  AND] 

THEN  LISTH  AFLD  NAME,  GCD  (PARKERSBURG),  JET  FUEL  CAP  n 

Ik.  RETRIEVE  AIRFIELDS  WITH  COUNTRY  =  USA,  NAVAID  FACILITY  =  TOWER  9 
AND  APPROACH  AND  VOR  AND  TACAN  AND  VORTAC  AND  RBN  AND  AIR/GND 
AND  DF  AND  GCA  AND  ILS ,  LTG  FACILITY  =  ANY  THEN  COUNT  “I 

15.  RETRIEVE  PERS  DEPLOY  STAT  WITH  SUM  BY  MDS  BY  POB  (TOTAL  ORG-AMN  11 

>1000),  CRTY/ STATE  =  USA;  ORG  KIND  =  RFG  THEN  LIST  SUM  =  INCR, 

PLAN  IDENT  -» 

16.  RETRIEVE  FORCE  STATUS  WITH  COMD  =  TAC,  UNIT  =  8lBW,  MDS  =  B52A,  21 
UNIT  LOC  =  PARKERSBURG  THEN  RETAIN  AND  LISTH  MIN  (ACFT  AUTH, 

CRWS  AUTH),  TITLE  =  WPSYS  AUTH:  RETRIEVE  PLAN  IDENT  WITH  MDS  = 
B52A,  CRWD  ACFT  RQ  <  [Rl,  WPSYS  AUTH,  AND]  THEN  LISTH  PLAN  IDENT *1 
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Sequence  and  Contents  of  the  Computer  Directed  Training  Sets 
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CUM  Y  SET  ..|Q 

INDEP  (R 
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CUM  Y  SET  ##|T 
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materials;  use  of  the  CDT  Exhibit  Book. 

More  detailed  coverage  on  the  use  of  the  console 
and  the  CDT  overlay. 

The  computer  component  and  the  input  and  output 
devices  of  the  473L  System;  the  file  struc¬ 
ture  and  the  types  of  data  storage  utilized 
by  System  473L;  basic  methods  of  data 
retrieval  with  System  473L;  more  on  the  use 
of  the  console  and  the  CDT  overlay. 

Basic  elements  of  a  QL  statement;  introduction 
to  Error  Messages;  use  of  the  AF  manuals 
covering  the  QL  data  base  and  system  vocabu¬ 
lary;  more  on  the  use  of  the  CDT  Exhibit  Book. 

More  on  the  use  of  the  AF  manuals  covering  the 
QL  data  base  and  system  vocabulary;  more  on 
the  use  of  the  CDT  Exhibit  Book;  more  on  the 
use  of  the  console  and  the  CDT  overlay. 
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Basic  instruction  on  writing  the  Output  Director 
and  Output  Selector 

Review  of  F  and  G 

Special  features  of  the  qualifier 

Special  features  of  the  selector 

Review  of  I  and  J 

GCD  functions 
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More  on  MIN  and  MAX  functions 

GREATEST  and  LEAST,  and  Review  of  MIN  and  MAX 

Review  of  0  and  P 

Simple  uses  of  SUM 

More  complex  uses  of  SUM 

Review  of  R  and  S 

Titles  and  Sorting  and  Compound  Qualifiers 

SAVE  Procedures  and  CHECK 

Review  of  U  and  V 

Complex  Queries 

Review  and  Practice  of  Sets  F  through  X, 
and  the  Posttest 
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